[SUPPORT] Compile Hudi 0.15 with Spark 3.5 and Scala 2.13

Describe the problem you faced
Compile Hudi 0.15 with Spark 3.5 and Scala 2.13
To Reproduce
[INFO] Reactor Summary for Hudi 0.15.0:
[INFO]
[INFO] Hudi ............................................... SUCCESS [  4.249 s]
[INFO] hudi-tests-common .................................. SUCCESS [  6.810 s]
[INFO] hudi-io ............................................ SUCCESS [ 11.286 s]
[INFO] hudi-common ........................................ SUCCESS [ 25.058 s]
[INFO] hudi-hadoop-common ................................. SUCCESS [  7.496 s]
[INFO] hudi-hadoop-mr ..................................... SKIPPED
[INFO] hudi-sync-common ................................... SKIPPED
[INFO] hudi-hive-sync ..................................... SKIPPED
[INFO] hudi-aws ........................................... SKIPPED
[INFO] hudi-timeline-service .............................. SKIPPED
[INFO] hudi-client ........................................ SUCCESS [  1.156 s]
[INFO] hudi-client-common ................................. SKIPPED
[INFO] hudi-spark-client .................................. SKIPPED
[INFO] hudi-spark-datasource .............................. SUCCESS [  1.155 s]
[INFO] hudi-spark-common_2.13 ............................. SKIPPED
[INFO] hudi-spark3-common ................................. SKIPPED
[INFO] hudi-spark3.2plus-common ........................... SKIPPED
[INFO] hudi-spark3.5.x_2.13 ............................... SKIPPED
[INFO] hudi-java-client ................................... SKIPPED
[INFO] hudi-spark_2.13 .................................... SKIPPED
[INFO] hudi-gcp ........................................... SKIPPED
[INFO] hudi-utilities_2.13 ................................ SKIPPED
[INFO] hudi-utilities-bundle_2.13 ......................... SKIPPED
[INFO] hudi-cli ........................................... SKIPPED
[INFO] hudi-flink-client .................................. SKIPPED
[INFO] hudi-datahub-sync .................................. SKIPPED
[INFO] hudi-adb-sync ...................................... SKIPPED
[INFO] hudi-sync .......................................... SUCCESS [  1.155 s]
[INFO] hudi-hadoop-mr-bundle .............................. SKIPPED
[INFO] hudi-datahub-sync-bundle ........................... SKIPPED
[INFO] hudi-hive-sync-bundle .............................. SKIPPED
[INFO] hudi-aws-bundle .................................... SKIPPED
[INFO] hudi-gcp-bundle .................................... SKIPPED
[INFO] hudi-spark3.5-bundle_2.13 .......................... SKIPPED
[INFO] hudi-presto-bundle ................................. SKIPPED
[INFO] hudi-utilities-slim-bundle_2.13 .................... SKIPPED
[INFO] hudi-timeline-server-bundle ........................ SKIPPED
[INFO] hudi-trino-bundle .................................. SKIPPED
[INFO] hudi-examples ...................................... SUCCESS [  1.155 s]
[INFO] hudi-examples-common ............................... SKIPPED
[INFO] hudi-examples-spark ................................ SKIPPED
[INFO] hudi-flink-datasource .............................. SUCCESS [  1.153 s]
[INFO] hudi-flink1.18.x ................................... FAILURE [  0.921 s]
[INFO] hudi-flink ......................................... SKIPPED
[INFO] hudi-examples-flink ................................ SKIPPED
[INFO] hudi-examples-java ................................. SKIPPED
[INFO] hudi-flink1.14.x ................................... FAILURE [  5.347 s]
[INFO] hudi-flink1.15.x ................................... FAILURE [  1.079 s]
[INFO] hudi-flink1.16.x ................................... FAILURE [  1.077 s]
[INFO] hudi-flink1.17.x ................................... FAILURE [  1.077 s]
[INFO] hudi-kafka-connect ................................. SKIPPED
[INFO] hudi-flink1.18-bundle .............................. SKIPPED
[INFO] hudi-kafka-connect-bundle .......................... SKIPPED
[INFO] hudi-cli-bundle_2.13 ............................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  56.950 s (Wall Clock)
[INFO] Finished at: 2024-08-27T09:16:39-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) on project hudi-flink1.18.x: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) on project hudi-flink1.17.x: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) on project hudi-flink1.16.x: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) on project hudi-flink1.15.x: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] Failed to execute goal on project hudi-flink1.14.x: Could not resolve dependencies for project org.apache.hudi:hudi-flink1.14.x:jar:0.15.0
[ERROR] dependency: org.apache.flink:flink-table-api-java-bridge_2.13:jar:1.14.5 (provided)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-api-java-bridge_2.13:jar:1.14.5 in Maven Central (https://repo.maven.apache.org/maven2)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-api-java-bridge_2.13:jar:1.14.5 in cloudera-repo-releases (https://repository.cloudera.com/artifactory/public/)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-api-java-bridge_2.13:jar:1.14.5 in confluent (https://packages.confluent.io/maven/)
[ERROR] 	org.apache.flink:flink-table-api-java-bridge_2.13:jar:1.14.5 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced
[ERROR] dependency: org.apache.flink:flink-table-runtime_2.13:jar:1.14.5 (provided)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-runtime_2.13:jar:1.14.5 in Maven Central (https://repo.maven.apache.org/maven2)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-runtime_2.13:jar:1.14.5 in cloudera-repo-releases (https://repository.cloudera.com/artifactory/public/)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-runtime_2.13:jar:1.14.5 in confluent (https://packages.confluent.io/maven/)
[ERROR] 	org.apache.flink:flink-table-runtime_2.13:jar:1.14.5 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced
[ERROR] dependency: org.apache.flink:flink-streaming-java_2.13:jar:1.14.5 (provided)
[ERROR] 	Could not find artifact org.apache.flink:flink-streaming-java_2.13:jar:1.14.5 in Maven Central (https://repo.maven.apache.org/maven2)
[ERROR] 	Could not find artifact org.apache.flink:flink-streaming-java_2.13:jar:1.14.5 in cloudera-repo-releases (https://repository.cloudera.com/artifactory/public/)
[ERROR] 	Could not find artifact org.apache.flink:flink-streaming-java_2.13:jar:1.14.5 in confluent (https://packages.confluent.io/maven/)
[ERROR] 	org.apache.flink:flink-streaming-java_2.13:jar:1.14.5 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced
[ERROR] dependency: org.apache.flink:flink-parquet_2.13:jar:1.14.5 (provided)
[ERROR] 	Could not find artifact org.apache.flink:flink-parquet_2.13:jar:1.14.5 in Maven Central (https://repo.maven.apache.org/maven2)
[ERROR] 	Could not find artifact org.apache.flink:flink-parquet_2.13:jar:1.14.5 in cloudera-repo-releases (https://repository.cloudera.com/artifactory/public/)
[ERROR] 	Could not find artifact org.apache.flink:flink-parquet_2.13:jar:1.14.5 in confluent (https://packages.confluent.io/maven/)
[ERROR] 	org.apache.flink:flink-parquet_2.13:jar:1.14.5 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced
[ERROR] dependency: org.apache.flink:flink-table-planner_2.13:jar:1.14.5 (provided)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-planner_2.13:jar:1.14.5 in Maven Central (https://repo.maven.apache.org/maven2)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-planner_2.13:jar:1.14.5 in cloudera-repo-releases (https://repository.cloudera.com/artifactory/public/)
[ERROR] 	Could not find artifact org.apache.flink:flink-table-planner_2.13:jar:1.14.5 in confluent (https://packages.confluent.io/maven/)
[ERROR] 	org.apache.flink:flink-table-planner_2.13:jar:1.14.5 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced
[ERROR]
[ERROR] -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hudi-flink1.18.x
albert@Alberts-MBP hudi % java -version
openjdk version "1.8.0_412"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_412-b08)
OpenJDK 64-Bit Server VM (Temurin)(build 25.412-b08, mixed mode)

Steps to reproduce the behavior:

mvn -T 2C clean package -DskipTests -Dspark3.5 -Dscala-2.13

Expected behavior
A clear and concise description of what you expected to happen.
Environment Description

Hudi version :

Spark version :

Hive version :

Hadoop version :

Storage (HDFS/S3/GCS..) :

Running on Docker? (yes/no) :

Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.

looking at https://mvnrepository.com/artifact/org.apache.flink/flink-table-api-java-bridge, there is no Scala v 2.13.  You need to skip the flink compiling.


QUser, QRelationship, QMicropost classes not found during compilation

Seems enhanced classes were developed to replace regular ones but never added to the repo
Commit 4f35f71b32c96b9afd2d697b03d32ec8e73c70fd (Jan-29) was the last compilable. All further refactorings are broken then

@alfishe
They will be automatically generated by annotation processor. If they were not generated, please try following.
mvn clean compile

Thanks,

@alfishe
I confirmed that it happens in importing new project in Intellij IDEA.

I'm afraid that it is a problem of IDEA. But here is a work around.
Do mvn clean compile
Comment out some lines (line 184~206 was OK) on pom.xml and import changes
Revert the pom.xml and import changes.
After that generated sources under target/generated-sources would be recognised by IDEA.


Running OpenEats without Docker

We should have a way to run OpenEats without docker. The following needs to be done in order for this to work.

[x] Adding non-docker support to openeats-web.
[x] Add documentation on how to setup openeats-web outside of docker.
[x] Adding non-docker support to openeats-api.
[x] Add documentation on how to setup openeats-api outside of docker.
[x] Because custom settings are stored in env files we need to make them available from a settings files for both openeats-api and openeats-web
[x] Add general documentation and an example of how an nginx configure would work
[x] Add general documentation oh how all the services are linked together.
[x] Help from others willing to test!

Feel free to add the link or the content to your docs.  If I forgot something from the wiki or it's not working
for some reason, don't hesitate to ask me about it.

Overall this works great! See below for my thoughts:

pip packages should be installed in a virtual env - see here
mysql-config wasn't found by default, it needs another packge - apt-get install libmysqlclient-dev
env vars - OpenEats-web only needs to have access to the NODE_ env vars and the API needs access to the others.

api:
# Database config
MYSQL_HOST=localhost
MYSQL_DATABASE=openeats
MYSQL_USER=username
MYSQL_ROOT_PASSWORD=password

# Django config
API_URL=<IP_ADDRESS>:5210
API_PORT=5210
DJANGO_SECRET_KEY=CHANGE_THIS:SECRETKEY
DJANGO_SETTINGS_MODULE=base.settings
DJANGO_DEBUG=False
ALLOWED_HOST=openeats.domain.com  # if you running from a domain you have to specify the hostname here [default value was localhost]

# If you are serving content behind an HTTPS proxy,
# Set this to `true`.
# See: https://docs.djangoproject.com/en/1.10/ref/settings/#secure-proxy-ssl-header
HTTP_X_FORWARDED_PROTO=true  # We are using ssl maybe you need to change it back to false

web:
export NODE_ENV=development
export NODE_URL=openeats.domain.com:5200 # As far as I tested the port can be anything here we wont use it
export NODE_API_URL=https://openeats.domain.com  # the openeats domain with http or https
export NODE_LOCALE=en

I you have the time and are willing, I would love a pull request with the docs in markdown. It would go in the docs folder and have a link on the main README. Otherwise, I will work on getting this into the project repo by next week :)

I was able to follow the above instructions / wiki link and successfully get openeats running on ubuntu without docker.  I did have to edit one additional file:  in the openeats-api repo, I had to edit  base/settings.py  lines 221 and 222 MEDIA_URL and STATIC_URL   and prepend my domain url in order to get recipe images to load successfully. (Prior to the edit, I could add an image to a recipe successfully, but not view it, as the image url would be http://localhost:5210/site-media/upload/recipe_photos/IMAGE_NAME.jpg instead of the correct http://openeats.example.com/site-media/upload/recipe_photos/IMAGE_NAME.jpg
--- settings.py.orig	2018-07-27 15:54:11.510476493 -0700
+++ settings.py	2018-07-27 15:54:26.146829183 -0700
@@ -216,8 +216,10 @@
 # URL that handles the media served from MEDIA_ROOT. Make sure to use a
 # trailing slash if there is a path component (optional in other cases).
 # Examples: "http://media.lawrence.com", "http://example.com/media/"
-MEDIA_URL = '/site-media/'
-STATIC_URL = '/static-files/'
+MEDIA_URL = 'http://openeats.example.com/site-media/'
+STATIC_URL = 'http://openeats.example.com/static-files/'

That will be dependent on your proxy server setup IE: Apache or Nginx should be handling this. This is documented with in the Wiki under /etc/apache2/sites-available/default.conf
For Nginx See:
https://github.com/open-eats/openeats-nginx/blob/master/default.conf#L38
For Apache See:
       Alias /static-files /opt/openeats/openeats-apache2/static-files
       <Directory "/opt/openeats/openeats-apache2/static-files">
           Options Indexes FollowSymLinks MultiViews
           AllowOverride All
           Order allow,deny
           allow from all
           Require all granted
       </Directory>
       Alias /site-media /opt/openeats/openeats-apache2/site-media
       <Directory "/opt/openeats/openeats-apache2/site-media">
           Options Indexes FollowSymLinks MultiViews
           AllowOverride All
           Order allow,deny
           allow from all
           Require all granted
       </Directory>

See: #44


Improving the plywood core development and HTML generation workflow

Current situation:
It's not convenient to tweak the HTML template right now, as one has to run generateHtml everytime one wants to see the new page. A build system might be useful to make this process easier
This issue is a follow-up to the conversation that happened by @KekOnTheWorld and @kolibril13 in https://github.com/kolibril13/plywood-gallery/issues/14#issuecomment-1357774633

If there was a tool (written in python) that allows basic preprocessing (Converting typescript to javascript, Bundle the JavaScript and CSS and minify), it would greatly improve the development Experience.
If we can't find such, we should consider using JS Type Documentation comments and then minifying the HTML after (removing unnessecary stuff like comments, whitespaces, ... to decrease the Bundle size and improve the Performance of the JS parser).


build: small fixes

fixes 'GOROOT' with space in path
fixes glob.sync breaking on windows path slash
fix makefile color printing

Found from getting this working on windows, the replaceAll is probably the only 'odd' thing, otherwise nothing should cause any issues on other platforms to my knowledge. I understand if you're not interested in supporting this, but with this i was able to 'finagle' native windows with msys2 to handle makefile command

awkward, quoting actually broke build. changed quote spot and changed echo -e to just printf, colors should work in terminal then locally and (hopefully) will also print in github actions correctly and without the -e


Strange validation_step and global_step behavior after every epoch

🐛 Bug

The following was produced with the BoringModel provided. With val_check_interval=10, last validation_step of first epoch was at global_step=59 and first validation_step of second epoch was at global_step=73, and so on. It seems like it is always off by len(dataset) % val_check_interval every time after an epoch. I think it is because the counter resets per epoch. In this case, the length of dataset is 64 and val_check_interval is 10. After the last validation_step in the first epoch at global_step=59, the first epoch ended at global_step=63, the counter resets and so the first validation_step of second epoch is at global_step=73.
Also, a while back I submitted an issue also related to val_check_interval. I think it would be nice to be able to log training_step and validation_step at the same global_step. My suggestion is to simply trigger validation_step when global_step % val_check_interval == 0, so in training_step we can manually trigger logging (for example, logging images) with this simple global_step % val_check_interval == 0 condition. This would greatly help debug models and plot graphs that training and validation logs are aligned. Thanks in advance :)
validation_step at global_step =  0
override any method to prove your bug
validation_step at global_step =  9
validation_step at global_step =  19
validation_step at global_step =  29
validation_step at global_step =  39
validation_step at global_step =  49
validation_step at global_step =  59
override any method to prove your bug
validation_step at global_step =  73
validation_step at global_step =  83
validation_step at global_step =  93
validation_step at global_step =  103
validation_step at global_step =  113
validation_step at global_step =  123
override any method to prove your bug
validation_step at global_step =  137
validation_step at global_step =  147
validation_step at global_step =  157
validation_step at global_step =  167
validation_step at global_step =  177
validation_step at global_step =  187
override any method to prove your bug
validation_step at global_step =  201
validation_step at global_step =  211
validation_step at global_step =  221
validation_step at global_step =  231
validation_step at global_step =  241
validation_step at global_step =  251
override any method to prove your bug
validation_step at global_step =  265
validation_step at global_step =  275
validation_step at global_step =  285
validation_step at global_step =  295
validation_step at global_step =  305
validation_step at global_step =  315

Please reproduce using the BoringModel and post here

# Copyright The PyTorch Lightning team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# --------------------------------------------
# --------------------------------------------
# --------------------------------------------
# USE THIS MODEL TO REPRODUCE A BUG YOU REPORT
# --------------------------------------------
# --------------------------------------------
# --------------------------------------------
import os
import torch
from torch.utils.data import Dataset
from pytorch_lightning import Trainer, LightningModule

class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):

    def __init__(self):
        """
        Testing PL Module
        Use as follows:
        - subclass
        - modify the behavior for what you want
        class TestModel(BaseTestModel):
            def training_step(...):
                # do your own thing
        or:
        model = BaseTestModel()
        model.training_epoch_end = None
        """
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def loss(self, batch, prediction):
        # An arbitrary loss to have a loss that updates the model weights during `Trainer.fit` calls
        return torch.nn.functional.mse_loss(prediction, torch.ones_like(prediction))

    def step(self, x):
        x = self.layer(x)
        out = torch.nn.functional.mse_loss(x, torch.ones_like(x))
        return out

    def training_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        return {"loss": loss}

    def training_step_end(self, training_step_outputs):
        return training_step_outputs

    def training_epoch_end(self, outputs) -> None:
        torch.stack([x["loss"] for x in outputs]).mean()

    def validation_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        return {"x": loss}

    def validation_epoch_end(self, outputs) -> None:
        torch.stack([x['x'] for x in outputs]).mean()

    def test_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        return {"y": loss}

    def test_epoch_end(self, outputs) -> None:
        torch.stack([x["y"] for x in outputs]).mean()

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(self.layer.parameters(), lr=0.1)
        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)
        return [optimizer], [lr_scheduler]

def run_test():
    class TestModel(BoringModel):

        def on_train_epoch_start(self) -> None:
            print('override any method to prove your bug')

        def validation_step(self, batch, batch_idx):
            output = self.layer(batch)
            loss = self.loss(batch, output)
            print("validation_step at global_step = ", self.global_step)
            return {"x": loss}

    # fake data
    train_data = torch.utils.data.DataLoader(RandomDataset(32, 64))
    val_data = torch.utils.data.DataLoader(RandomDataset(32, 64))
    test_data = torch.utils.data.DataLoader(RandomDataset(32, 64))

    # model
    model = TestModel()
    trainer = Trainer(
        default_root_dir=os.getcwd(),
        # limit_train_batches=1,
        limit_val_batches=1,
        val_check_interval=10,
        max_epochs=5,
        weights_summary=None,
    )
    trainer.fit(model, train_data, val_data)
    trainer.test(test_dataloaders=test_data)

if __name__ == '__main__':
    run_test()

To Reproduce

Expected behavior

validation_step at global_step =  0
override any method to prove your bug
validation_step at global_step =  9
validation_step at global_step =  19
validation_step at global_step =  29
validation_step at global_step =  39
validation_step at global_step =  49
validation_step at global_step =  59
override any method to prove your bug
validation_step at global_step =  69
validation_step at global_step =  79
validation_step at global_step =  89
validation_step at global_step =  99
validation_step at global_step =  109
validation_step at global_step =  119
...

Environment
Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).
You can get the script and run it with:
wget https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
# For security purposes, please check the contents of collect_env_details.py before running it.
python collect_env_details.py

CUDA:
- GPU:
- Tesla T4
- available:         True
- version:           10.2
Packages:
- numpy:             1.19.1
- pyTorch_debug:     False
- pyTorch_version:   1.6.0
- pytorch-lightning: 1.0.1
- tqdm:              4.50.1
System:
- OS:                Linux
- architecture:
- 64bit
- ELF
- processor:         x86_64
- python:            3.8.5
- version:           #25~18.04.1-Ubuntu SMP Fri Sep 11 21:07:02 UTC 2020

Additional context

@teddykoker any update?

@tchaton can you take a look to see if there's a bug here?

@edenlightning I can confirm this is not a bug, val_check_interval works within each epoch and resets every epoch.

Reseting every epoch makes some sense when the val_check_interval is a float between 0 and 1 but not when val_check_interval is an int. However, there is still a problem that we are not able to align the training step and validation step logging.
# DEFAULT
trainer = Trainer(val_check_interval=0.95)

# check every .25 of an epoch
trainer = Trainer(val_check_interval=0.25)

# check every 100 train batches (ie: for `IterableDatasets` or fixed frequency)
trainer = Trainer(val_check_interval=100)

Code
# Copyright The PyTorch Lightning team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# --------------------------------------------
# --------------------------------------------
# --------------------------------------------
# USE THIS MODEL TO REPRODUCE A BUG YOU REPORT
# --------------------------------------------
# --------------------------------------------
# --------------------------------------------
import os
import torch
from torch.utils.data import Dataset
from pytorch_lightning import Trainer, LightningModule

class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):

    def __init__(self):
        """
        Testing PL Module
        Use as follows:
        - subclass
        - modify the behavior for what you want
        class TestModel(BaseTestModel):
            def training_step(...):
                # do your own thing
        or:
        model = BaseTestModel()
        model.training_epoch_end = None
        """
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def loss(self, batch, prediction):
        # An arbitrary loss to have a loss that updates the model weights during `Trainer.fit` calls
        return torch.nn.functional.mse_loss(prediction, torch.ones_like(prediction))

    def step(self, x):
        x = self.layer(x)
        out = torch.nn.functional.mse_loss(x, torch.ones_like(x))
        return out

    def training_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        return {"loss": loss}

    def training_step_end(self, training_step_outputs):
        return training_step_outputs

    def training_epoch_end(self, outputs) -> None:
        torch.stack([x["loss"] for x in outputs]).mean()

    def validation_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        return {"x": loss}

    def validation_epoch_end(self, outputs) -> None:
        torch.stack([x['x'] for x in outputs]).mean()

    def test_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        return {"y": loss}

    def test_epoch_end(self, outputs) -> None:
        torch.stack([x["y"] for x in outputs]).mean()

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(self.layer.parameters(), lr=0.1)
        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)
        return [optimizer], [lr_scheduler]

from pytorch_lightning.loggers import WandbLogger
def run_test():
    class TestModel(BoringModel):

        def on_train_epoch_start(self) -> None:
            print('override any method to prove your bug')

        def validation_step(self, batch, batch_idx):
            output = self.layer(batch)
            loss = self.loss(batch, output)
            print("validation_step at global_step = ", self.global_step)
            return {"x": loss}

    # fake data
    train_data = torch.utils.data.DataLoader(RandomDataset(32, 64))
    val_data = torch.utils.data.DataLoader(RandomDataset(32, 64))
    test_data = torch.utils.data.DataLoader(RandomDataset(32, 64))

    # model
    model = TestModel()
    trainer = Trainer(
        default_root_dir=os.getcwd(),
        # limit_train_batches=1,
        limit_val_batches=1,
        val_check_interval=10,
        max_epochs=5,
        weights_summary=None,
        logger=WandbLogger(
            save_dir="./",
            project="BoringModel",
        )
    )
    trainer.fit(model, train_data, val_data)
    trainer.test(test_dataloaders=test_data)

if __name__ == '__main__':
    run_test()

After adding WandbLogger to the above code, the global_step is no longer an integer.
Epoch 0:   0%|          | 0/70 [00:00<?, ?it/s] override any method to prove your bug
Epoch 0:  14%|█▍        | 10/7validation_step at global_step =  9282, v_num=6sdy]
Epoch 0:  30%|███       | 21/7validation_step at global_step =  1945, v_num=6sdy]
Epoch 0:  46%|████▌     | 32/7validation_step at global_step =  2948, v_num=6sdy]
Epoch 0:  61%|██████▏   | 43/7validation_step at global_step =  39714, v_num=6sdy]
Epoch 0:  77%|███████▋  | 54/7validation_step at global_step =  49764, v_num=6sdy]
Epoch 0:  93%|█████████▎| 65/7validation_step at global_step =  597.785, v_num=6sdy]
Epoch 1:   0%|          | 0/70 [00:00<?, ?it/s, loss=1860.793, v_num=6sdy]override any method to prove your bug
Epoch 1:  14%|█▍        | 10/7validation_step at global_step =  739.871, v_num=6sdy]
Epoch 1:  30%|███       | 21/7validation_step at global_step =  831.552, v_num=6sdy]
Epoch 1:  46%|████▌     | 32/7validation_step at global_step =  933.020, v_num=6sdy]
Epoch 1:  61%|██████▏   | 43/7validation_step at global_step =  103.344, v_num=6sdy]
Epoch 1:  77%|███████▋  | 54/7validation_step at global_step =  113.061, v_num=6sdy]
Epoch 1:  93%|█████████▎| 65/7validation_step at global_step =  123.813, v_num=6sdy]
Epoch 2:   0%|          | 0/70 [00:00<?, ?it/s, loss=6482.896, v_num=6sdy]override any method to prove your bug
Epoch 2:  14%|█▍        | 10/7validation_step at global_step =  137.802, v_num=6sdy]
Epoch 2:  30%|███       | 21/7validation_step at global_step =  147325, v_num=6sdy]
Epoch 2:  46%|████▌     | 32/7validation_step at global_step =  157170, v_num=6sdy]
Epoch 2:  61%|██████▏   | 43/7validation_step at global_step =  167899, v_num=6sdy]
Epoch 2:  77%|███████▋  | 54/7validation_step at global_step =  177695, v_num=6sdy]
Epoch 2:  93%|█████████▎| 65/7validation_step at global_step =  187.520, v_num=6sdy]
Epoch 3:   0%|          | 0/70 [00:00<?, ?it/s, loss=1661.758, v_num=6sdy]override any method to prove your bug
Epoch 3:  14%|█▍        | 10/7validation_step at global_step =  201.825, v_num=6sdy]
Epoch 3:  30%|███       | 21/7validation_step at global_step =  211465, v_num=6sdy]
Epoch 3:  46%|████▌     | 32/7validation_step at global_step =  221669, v_num=6sdy]
Epoch 3:  61%|██████▏   | 43/7validation_step at global_step =  231428, v_num=6sdy]
Epoch 3:  77%|███████▋  | 54/7validation_step at global_step =  241890, v_num=6sdy]
Epoch 3:  93%|█████████▎| 65/7validation_step at global_step =  251.060, v_num=6sdy]
Epoch 4:   0%|          | 0/70 [00:00<?, ?it/s, loss=1521.544, v_num=6sdy]override any method to prove your bug
Epoch 4:  14%|█▍        | 10/7validation_step at global_step =  265.673, v_num=6sdy]
Epoch 4:  30%|███       | 21/7validation_step at global_step =  275279, v_num=6sdy]
Epoch 4:  46%|████▌     | 32/7validation_step at global_step =  285370, v_num=6sdy]
Epoch 4:  61%|██████▏   | 43/7validation_step at global_step =  295927, v_num=6sdy]
Epoch 4:  77%|███████▋  | 54/7validation_step at global_step =  305516, v_num=6sdy]
Epoch 4:  93%|█████████▎| 65/7validation_step at global_step =  315.920, v_num=6sdy]
Epoch 4: 100%|██████████| 70/70 [00:00<00:00, 568.51it/s, loss=1508.531, v_num=6sdy]

@tchaton thoughts?

Oh wait sorry, scratch that. I think it was just the printing made me think the global_steps were not integers. My bad.
But I still persist my suggestion to trigger validation_step when global_step % val_check_interval == 0 for self.log(on_step=True).  :D


Fix req without directory separator ending

I use DEBUG to show the koa-serve debug info : DEBUG=koa-serve node index.js
And I saw following debug info and I've fixed it in the source code
GET /game_1049911 - 2
  koa-serve requested: +8s /game_1049911
  koa-serve served: +1ms /home/jun1st/yongdudehekou/game_1049911index.html

Meanwhile I used Assert to pass all tests including the extra one added and bump the version to v0.1.8

Ah, I found it not a elegant and proper mean to solve the problem
for the browser will solve <script type="text/javascript" src="js/createjs.min.js"></script> to be http://www.example.com/js/createjs.min.js instead of http://www.example.com/some_path/js/createjs.min.js if you visit http://www.exmaple.com/some_path


Any toolbar with external buttons ?

Would you have a toolbar (for example in the style of http://summernote.org) which would be easy to use with handsontable?
Thanks!

Thanks for your comment!
As far as I am concerned, for cells formatting, there isn't yet a straight API to do so. You would need a special renderer function for each individual cells and it could be very app-specific.
There is already API that does merging, undo, redo, etc. so hopefully in the future there would be more coming up.
Adding extra feature Handsontable does not yet have is little beyond the scope of this package.
I would consider developing this toolbar into a separate plugin in another repo,
and then make it depend on this package in the future.
I am actually planning to do something similar to what you have suggested. (As as separate plugin)
But it is still in development list, I will release it when it is ready. :D

Sure, I didn't mean to include a toolbar in this package. I was just asking if you knew something that could be used without re-doing everything (maybe some icons related to spreadsheet editing ?).
I think it would be good to have at least a minimum toolbar that could replace the functions of the context menu (which is not available on touch devices - as far as I know).
So I'm going to try to do something and I can keep you informed if you are interested.

Sure!
Would love to see handsontable to be fully touch device capable!

@jm4r7in, any luck about this matter? Did you do any progress?


Session with microros agent is constantly re-established

Issue template

Hardware description: STM32F407
RTOS: FreeRTOS
Installation type: STM32CubeIDE, micro_ros_stm32cubemx_utils
Version or commit hash: galactic

Steps to reproduce the issue
Follow micro_ros_stm32cubemx_utils. I'm using UART and DMA approach. From PC side I'm using docker run -it --rm --net=host -v /dev:/dev --privileged microros/micro-ros-agent:galactic serial --dev /dev/ttyUSB0 -b 115200 -v6
Expected behavior
Handshake between MCU and MicroROS agent. Discover topics on ROS2.
Actual behavior
Session is constantly re-established. DDS log never comes up. Serial data seems to be sent and received. (Log added in Additional information).
Additional information
This behaviour is constantly repeated through time, this is only the first lines. In MCU side, code hangs in rclc_support_init().

If I use minicom I can see MCU sending data at 115200b.

I don't know the reason of this behaviour (I don't have much knowledge of DDS layer)
Any help would be appreciated, thanks!

@Acuadros95 can you take a look?

Hi @leoborgnino:

Are you using minicom at the same time as the agent?
Can you start the Agent before the board is connected? In rclc_support_init() The MCU will try to connect with the Agent for 10 seconds before it gives up. This could lead to your problem is the agent is not available when the board is connected.
You can use the ping functionality to avoid this, check this example: micro-ros_reconnection_example

Hi @leoborgnino:

Are you using minicom at the same time as the agent?
Can you start the Agent before the board is connected? In rclc_support_init() The MCU will try to connect with the Agent for 10 seconds before it gives up. This could lead to your problem is the agent is not available when the board is connected.
You can use the rmw_uros_ping_agent functionality to avoid this, check this example: micro-ros_reconnection_example

Hi Pablo and Antonio, thanks for the fast reply!

I only used minicom to test if MCU was sending data at 115200b but I was not using it with the agent.
I start the agent and then I connect the MCU, but I got the same behaviour
If I try rmw_uros_ping_agent(100,1) function, code hangs there with no log in agent. If I open minicom I see MCU sending data.

Could you share your code, so we can replicate on our side?

Could you share your code, so we can replicate on our side?

Hi Antonio,
The code is in: https://github.com/leoborgnino/microros-stm32f407
Steps (I think you guys know this but can be an error for my side in the steps):

clone the repo
in repo directory: git clone https://github.com/micro-ROS/micro_ros_stm32cubemx_utils
docker pull microros/micro_ros_static_library_builder:foxy && docker run -it --rm -v $(pwd):/project --env MICROROS_LIBRARY_FOLDER=micro_ros_stm32cubemx_utils/microros_static_library microros/micro_ros_static_library_builder:foxy
make in root folder generates .elf
I use CubeIDE to import elf and debug
docker run -it --rm --net=host -v /dev:/dev --privileged microros/micro-ros-agent:foxy serial --dev /dev/ttyUSB0 -b 115200 -v6

Thanks for the help!

Hi, I have just tested the DMA transport on a F746 and its looking good.
Some ideas based on your instructions:

You are missing this step Set the DMA priotity to Very High for Tx and Rx
Watch-out for branches, you are using foxy for the builder and agent, but the main branch of micro_ros_stm32cubemx_utils is humble
Did you try the regular IT transport? To discard problems on the Agent side.
Also try with the repo sample_main app.
This PR should help with the session re-established problem, rebuild the library once its merged:
Fix wait_session_status listen timeout eProsima/Micro-XRCE-DDS-Client#322

Please check this points and give feedback. If the problem persist, give info about the stm Firmware Version you are using. Example (Project Manager tab on CubeIDE):

Hi Antonio, I'm off on holidays until 12 Jul. I will test that and write you back, thanks!

Closing, reopen if the problem persist when you come back!

I hv the same issue any idea how to fix it?

I hv the same issue any idea how to fix it?

I reinstalled everything in another machine, and it worked, so I’m not really sure what the issue was.

it was working previously for me. Then after reinstallation of ubuntu it stopped working..


Error: Could not activate connection - When wrong context is set.

What happened?
I tried to run gefyra con connect -n XXX while having an active k8s context which does not respond to the cluster I was trying to connect to. I got he following error:
"Error: Could not activate connection"
What did you expect to happen?
I was expecting that an existing gefyra connection (which was stopped) is aware of the k8s context and that it does not matter which context I am in currently. But maybe this was/is a wrong assumption.
Please provide the output of gefyra check.

How can we reproduce it (as minimally and precisely as possible)?
Create a connection to a cluster.
Stop that connection.
Change the k8s context.
Try to connect again.
What Kubernetes setup are you working with?

$ kubectl version
# paste output here

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Anything else we need to know?
No response

Right now it worked fine (I could "gefyra con connect -n ..." to a cluster while having a different context). So I am not sure what exactly triggered the error reported here.

This was reworked in 2.1.0


JSDOM environement: blob is not typeof Blob

Describe the bug
When using node 18, there appears to be two instances of the Blob class when running tests inside a JSDOM environment.
This is very similar to a previous issue, #1703.
Reproduction
Here's a minimal repo that reproduces the error: https://github.com/aaronklinker-st/vitest-blob-issue
When running the following test:
/**
 * @vitest-environment jsdom
 */
import { JSDOM } from "jsdom";
import { expect, test } from "vitest";

function stringToDataUrl(str, BlobClass) {
  return new Promise((res) => {
    const blob = new BlobClass([str], { type: "plain/text" });
    const reader = new FileReader();
    reader.onload = (e) => res(e.target.result);
    reader.readAsDataURL(blob);
  });
}

test("Fails", async () => {
  const actual = await stringToDataUrl("hello world", Blob);
  expect(actual).toEqual("data:plain/text;base64,aGVsbG8gd29ybGQ=");
});

test("Passes", async () => {
  const dom = new JSDOM();
  const actual = await stringToDataUrl("hello world", dom.window.Blob);
  expect(actual).toEqual("data:plain/text;base64,aGVsbG8gd29ybGQ=");
});

It fails when using the global Blob, but works when specifying a JSDOM's Blob class instead.
Here's the test output:
$ npm test run

> test
> vitest run

 RUN  v0.23.4 /Users/aklinker1/Development/Temp Repos/vitest-blob-error

 ❯ example.test.js (2)
   × Fails
   ✓ Passes

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 FAIL  example.test.js > Fails
TypeError: Failed to execute 'readAsDataURL' on 'FileReader': parameter 1 is not of type 'Blob'.
 ❯ exports.convert node_modules/jsdom/lib/jsdom/living/generated/Blob.js:22:9
 ❯ FileReader.readAsDataURL node_modules/jsdom/lib/jsdom/living/generated/FileReader.js:197:23
 ❯ example.test.js:12:12
     10|     const reader = new FileReader();
     11|     reader.onload = (e) => res(e.target.result);
     12|     reader.readAsDataURL(blob);
       |            ^
     13|   });
     14| }
 ❯ stringToDataUrl example.test.js:8:10
 ❯ example.test.js:17:24

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: {
  "__defineGetter__": "Function<__defineGetter__>",
  "__defineSetter__": "Function<__defineSetter__>",
  "__lookupGetter__": "Function<__lookupGetter__>",
  "__lookupSetter__": "Function<__lookupSetter__>",
  "__proto__": null,
  "hasOwnProperty": "Function<hasOwnProperty>",
  "isPrototypeOf": "Function<isPrototypeOf>",
  "propertyIsEnumerable": "Function<propertyIsEnumerable>",
  "toLocaleString": "Function<toLocaleString>",
  "valueOf": "Function<valueOf>",
}

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

Test Files  1 failed (1)
     Tests  1 failed | 1 passed (2)
  Start at  10:04:34
  Duration  1.31s (transform 356ms, setup 0ms, collect 19ms, tests 12ms)

As you can see, this is quite the confusing error message since blob is definitely an instance of the Blob class, meaning there are two Blob classes available, but only one is accepted by the readAsDataURL method.
This passes in Node 16, likely because node 18 added it's own Blob global, similar to #1703's ArrayBuffer
System Info
System:
    OS: macOS 12.6
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 10.21 GB / 32.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.9.0 - ~/.nvm/versions/node/v18.9.0/bin/node
    npm: 8.19.1 - ~/.nvm/versions/node/v18.9.0/bin/npm
  Browsers:
    Chrome: 105.0.5195.125
    Firefox: 103.0.1
    Firefox Developer Edition: 106.0
    Safari: 16.0
  npmPackages:
    vitest: ^0.23.4 => 0.23.4

Used Package Manager
npm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

🎉 🎉 Thanks! Easy change


Error tagged bam

Hello @BGSpiegl ,
Thanks for developing this great tool. I get the output without errors but if I try to use the -ob parameter to get the tagged bam file, I get the following error (I tried with several different samples):
Traceback (most recent call last):   File "miniconda3/envs/GCparagon/bin/gcparagon", line 8, in <module>     sys.exit(main())   File "miniconda3/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 3174, in main     tag_bam_with_correction_weights_parallel(   File "miniconda3/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 2304, in tag_bam_with_correction_weights_parallel     tagged_scaffold_bam_files_in_order = order_bams(bam_list=tagged_scaffold_bam_files)   File "miniconda3/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 2163, in order_bams     chrm, strt, stp = Path(bam_nm).name.split('.GCcorr')[0].split('+')[-1].split('-')  # chrom-start-stop ValueError: too many values to unpack (expected 3) Continuing to process next BAM file..
What could this be due to? Thank you.
Regards

Hi Antonio. Thank you for submitting the issue. I am sorry that you experience problems with GCparagon.
A question form my side: what was the name of your input BAM file?
GCparagon uses multiprocessing to tag alignments from different scaffolds simultaneously.
Scaffolds are subdivided in 50,000,000 Mb genomic intervals.
To avoid scrambling up the output, each process writes a separate output file inside the temporary directory, one per genomic interval at a time.
These tagging output files include a scaffold tag like this:
BAMNAME+chr1-0-50000000.GCcorr.bam
where +chr1-0-50000000 identifies the processed genomic interval.
After tagging, the parent process gathers the tagged BAM file paths and concatenates them according to the scaffold order of the BAM header and the coordinates of the genomic intervals.
I ran a test sample using a slightly updated code where the troublemaker-piece of code from your stack trace tries to obtain this information from the tagged BAM file name: (warning-pseudocode)

extract chrm, strt, stp from the BAM file name by
splitting <BAMNAME+chr1-0-50000000.GCcorr.bam> based on substring '.GCcorr', taking the first element
split again at character '+', taking the last element (e.g., chr1-0-50000000)
split that again at characters - which should yield the scaffold name and the genomic coordinates 'chr1',0 and 50000000

This will fail if your BAM name contains multiple instances of the substring .GCcorr. So make sure this is not the case.
I changed now splitting to substring .GCcorr.bam which should work better.
Please clone the latest version v0.6.4 and retry!

Thanks for the quick response and the attempt to solve the problem, which seems to have been solved, but now I am getting another error; at first, my bam file name was ICE22_recal.bam, I even tried renaming it to ICE22.bam but I get the following error:
| 2024-01-24 23:17:20,129 - GCparagon - ERROR - The following exception occurred when trying to process BAM file ICE22.bam:
Traceback (most recent call last):
File "miniconda3/envs/GCparagon/bin/gcparagon", line 8, in
sys.exit(main())
File "miniconda3/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 3174, in main
tag_bam_with_correction_weights_parallel(
File "miniconda3/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 2304, in tag_bam_with_correction_weights_parallel
tagged_scaffold_bam_files_in_order = order_bams(bam_list=tagged_scaffold_bam_files)
File "miniconda3/envs/GCparagon/lib/python3.10/site-packages/GCparagon/correct_GC_bias.py", line 2164, in order_bams
bam_paths_with_locus.append(((chrm, int(strt)), str(bam_nm)))
ValueError: invalid literal for int() with base 10: 'A*01:02'
Continuing to process next BAM file..
Thanks.

It seems there is a timestamp in the name of one of your tagged output scaffold BAM files for at least one scaffold inside your temporary directory. Can you pls have a look how these scaffold BAM files are named inside your temporary directory? If you did not specify a temporary directory (which you should definitely do), the otemporary output is likely written to your system temp under /tmp/<sample_identifier>/
I have problems reproducing your error though.
As a first means of troubleshooting, please try adding the following piece of code (with same indentation as line 2164) after line 2164 in the script correct_GC_bias.py:
print(f"{chrm}, {strt}, {stp}")
I ran the v0.6.4 code in --tag-only mode to see if there might be a problem. There was none for me.
these were my command line parameters:
--bam /some_paht_to/INPUT.bam --correction-weights /another_path_to/INPUT_gc_weights_4simsMean.2IQRoutliersRemoved.2IgaussSmoothed.txt.gz --tag-only --output-unaligned-reads --tag-name TG --temporary-directory /mnt/scratch_drive/tmp
I need to ask you to provide the following details so that I can understand your situation better:

your complete command line call to GCparagon
the OS you are working on
the hardware you are using, specifically, the temporary directory and how many cores you are using for parallel processing (e.g., is any of your paths on a remote file system?)
are you using any custom-made files? (e.g., are you working with a non-human sample/reference genome)
did you clone the v0.6.4 repository anew or were you using a patchwork of extended or adapted code? If not, please clone the dev branch and try again!
have you defined the variable annotation_str anywhere in the code?

I encountered a problem where repeatedly outputting to the same directory fails because the output file already exists.
I changed that behavior to delete any preexisting output files before attempting to move files from the temporary directory there.
Thank you!


Sumo-DataminrPulse: First Checkin

PR Details
This PR includes the necessary files for the new Dataminr Pulse App for Sumologic.
Dataminr Pulse’s AI-powered real-time intelligence brings faster detection and response into Sumo Logic Cloud Security Analytics. Learn about threats as they unfold to prioritize and manage risk, and protect your physical and digital assets in real time. Dataminr Pulse for Cyber Risk easily fits into your Sumo Logic Cloud Security Analytics workflows, enabling rapid identification and mitigation of emerging threats so you can deliver faster time to detection and response. The global leader in AI for risk detection since 2009, Dataminr Pulse is relied on by two-thirds of Fortune 100 companies to inform their physical, cyber, and converged security operations.
We have three dashboards for this app namely:

Dataminr - Alerts Overview
Dataminr - Alerts Drilldown
Dataminr - Cyber Threat Landscape

Description
The PR includes:

App JSON file - exported JSON code with the utmost best practices mentioned in this doc (https://docs.google.com/document/d/1xfYfruru0RFWOH23GRrRzosSlepJUTcakLFHLHdpEtc/edit?usp=sharing).
Manifest file: With all the necessary details. The helpURL, iconURL, and screenshot URLs are yet to be updated.
Readme file: Created using this template(https://docs.google.com/document/d/1Ae9nG--GXo0nkdCNMZIRZIuhCXwJyHvvAwe2Upc6kRo/edit).
Sample log file.
Screenshots: High-resolution screenshots of the three dashboards.
Dataminr Pulse app icon.

Checklist

[x] Changes follow the best practices from the given docs.
[ ] Updated CHANGELOG.md.
[x] Ran unit tests locally.
[x] Updated screenshots/sample logs if any.

@parthCDSystems  if you could address above feedbacks in couple of days we can release this app with Friday's release.

LGTM


Paper dropdown menus near the bottom of the window should drop up

Paper dropdowns near the bottom of the window always open down, creating a compressed scroll area that's very difficult to navigate. (This is doubly challenging on OSX, wherein scroll bars are invisible by default.)
A video illustrates this issue in the demo page --
https://drive.google.com/file/d/0B8dway2WcUoSNWZrcFc3LVI0aGM/view?usp=sharing
I'd expect the behavior of polyer dropdowns to match other UI dropdown widgets and drop 'up' rather than down if the dropdown is too close to the bottom of the window and has more room above than below.

+1!
On Oct 23, 2015 2:17 PM, "Michael Piatek"<EMAIL_ADDRESS>wrote:

Paper dropdowns near the bottom of the window always open down, creating a
compressed scroll area that's very difficult to navigate. (This is doubly
challenging on OSX, wherein scroll bars are invisible by default.)
A video illustrates this issue in the demo page --
https://drive.google.com/file/d/0B8dway2WcUoSNWZrcFc3LVI0aGM/view?usp=sharing
I'd expect the behavior of polyer dropdowns to match other UI dropdown
widgets and drop 'up' rather than down if the dropdown is too close to the
bottom of the window and has more room above than below.
—
Reply to this email directly or view it on GitHub
https://github.com/Polymer/polymer/issues/2616.

This issue was moved to PolymerElements/paper-dropdown-menu#85


Question about the cls token

Awesome work of introducing transformer into ReID! But maybe I don't totaly understand the paper, having some confusion about the cls token. As shown in your paper, the cls token is a learnable embedding. And the code just initialize it as a parameter with normal distribution, and expand it to match the examples' shape in every batch. That is to say, the cls's value is no difference for each data when the model has completed training, Right? So I don't understand the meaning of setting it...
Thanks a lot for your help!

You can refer to BERT and ViT to better understand the [CLS] token. The [CLS] token is similar with the parameters of the center loss. We need give it a value as the initial value. But the value of the [CLS] token is related to Q,K,V, which are different for different data after the model inference. Therefore the [CLS] token is different for each data when the model has completed training.

Ohhh! I get it now. Thanks so much for your kindly reply!!


Portainer Edge Agent 2.14.0 unable to connect to Portainer

Bug description
Upgrade of Portainer to 2.14.0 went fine. After upgrading the Edge Agents from 2.13.1 to 2.14.0 they're unable to connect to Portainer. I run Portainer behind an AWS ALB. The setup is like the nginx reverse proxy setup described in the documentation.
Expected behavior
Edge Agents from 2.14.0 are able to connect to Portainer.
Portainer Logs
portainer/agent:2.14.0: Fail :x:
2022/07/01 09:15:38 [INFO] [main] [message: Agent running on Docker platform]
2022/07/01 09:15:38 [INFO] [edge] [message: Edge key loaded from options]
2022/07/01 09:15:38 [INFO] [edge,registry] [message: Starting registry credential server]
2022/07/01 09:15:38 [INFO] [http] [server_addr: 172.X.X.X] [server_port: 9001] [use_tls: false] [api_version: 2.14.0] [message: Starting Agent API server]
2022/07/01 09:15:43 [ERROR] [edge] [message: an error occured during short poll] [error: Get "https://$FQDN:9443/api/endpoints/37/edge/status": tls: server selected unsupported protocol version 303]
2022/07/01 09:15:48 [ERROR] [edge] [message: an error occured during short poll] [error: Get "https://$FQDN:9443/api/endpoints/37/edge/status": tls: server selected unsupported protocol version 303]
2022/07/01 09:15:53 [ERROR] [edge] [message: an error occured during short poll] [error: Get "https://$FQDN:9443/api/endpoints/37/edge/status": tls: server selected unsupported protocol version 303]
2022/07/01 09:15:58 [ERROR] [edge] [message: an error occured during short poll] [error: Get "https://$FQDN:9443/api/endpoints/37/edge/status": tls: server selected unsupported protocol version 303]
2022/07/01 09:16:03 [ERROR] [edge] [message: an error occured during short poll] [error: Get "https://$FQDN:9443/api/endpoints/37/edge/status": tls: server selected unsupported protocol version 303]

portainer/agent:2.13.1: Works :heavy_check_mark:
2022/07/01 09:20:37 [INFO] [main] [message: Agent running on Docker platform]
2022/07/01 09:20:37 [INFO] [edge] [message: Edge key loaded from options]
2022/07/01 09:20:37 [INFO] [edge,registry] [message: Starting registry credential server]
2022/07/01 09:20:37 [INFO] [http] [server_addr: 172.X.X.X] [server_port: 9001] [secured: false] [api_version: 2.13.1] [message: Starting Agent API server]
2022/07/01 09:21:03 client: Connecting to ws://$FQDN:8000
2022/07/01 09:21:03 client: Specified deprecated MD5 fingerprint (69:60:66:57:60:55:b9:10:4e:d8:66:bc:71:32:b7:18), please update to the new SHA256 fingerprint: 1866XqG96O1rxnEdNlnbFY/XcVrwQ7ONVKOCX9KZOEs=
2022/07/01 09:21:03 client: Connected (Latency 23.114275ms)

portainer/agent:2.13.0: Works :heavy_check_mark:
2022/07/01 09:26:23 [INFO] [main] [message: Agent running on Docker platform]
2022/07/01 09:26:23 [INFO] [edge] [message: Edge key loaded from options]
2022/07/01 09:26:23 [INFO] [edge,registry] [message: Starting registry credential server]
2022/07/01 09:26:23 [INFO] [http] [server_addr: 172.X.X.X] [server_port: 9001] [secured: false] [api_version: 2.13.0] [message: Starting Agent API server]
2022/07/01 09:27:03 client: Connecting to ws://$FQDN:8000
2022/07/01 09:27:03 client: Specified deprecated MD5 fingerprint (69:60:66:57:60:55:b9:10:4e:d8:66:bc:71:32:b7:18), please update to the new SHA256 fingerprint: 1866XqG96O1rxnEdNlnbFY/XcVrwQ7ONVKOCX9KZOEs=
2022/07/01 09:27:03 client: Connected (Latency 23.924627ms)

curl
curl -v https://$FQDN:9443/api/endpoints/28/edge/status
*   Trying $IP:9443...
* TCP_NODELAY set
* Connected to $FQDN ($IP) port 9443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=$FQDN
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ef2d5bf2f0)
> GET /api/endpoints/28/edge/status HTTP/2
> Host: $FQDN:9443
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 403
< date: Fri, 01 Jul 2022 09:23:05 GMT
< content-type: application/json
< content-length: 90
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
<
{"message":"Permission denied to access environment","details":"missing Edge identifier"}
* Connection #0 to host $FQDN left intact

Steps to reproduce the issue:

Run Portainer 2.14.0 behind an AWS ALB
Update Edge Agents from 2.13.1 to 2.14.0
Try to connect to the Edge Agent Environment
Connection fails
Check Edge Agent logs for errors

Technical details:

Portainer version: 2.14.0
Docker version (managed by Portainer): 20.10.17
Platform (windows/linux): linux
Command used to start Portainer (docker run -p 9443:9443 portainer/portainer): Portainer behind AWS ALB reverse proxy which supports TLS 1.2
Browser: Chrome (stable)
Use Case (delete as appropriate):Using Portainer in a Commercial setup.
Have you reviewed our technical documentation and knowledge base? Yes

Additional context
Might be related to #6902

Same issue here but I'm running Portainer behind Traefik. Downgrading the edge agent to v2.13.1 mitigated the issue (for now)…

Confirmed, investigation underway.
Thank you.


clean yum cache in rhel8 based s2i-core

use yum instead of dnf as yum is now installed in the base image by default

rhel8 failure is from trying to build centos7 based cassandra for some reason, will investigate

rhel7 failure is from explicit shadow-utils install and rpm verification scheck even though it comes from base image. Removing in images where it is causing issues in separate PRs.

[test]

centos7 failure should be fixed by https://github.com/sclorg/mysql-container/commit/0ad545c7e1e8b4eccf5753b24f11f84a23ac1f90 but it seems the CI did not re-trigger

LGTM. Thanks. I think it can be merged even w/o CI tests.

CI tests green now


Issues with Padding missing font?

This is how the find and replace looks:

Buttons are missing here too, something to do with the padding I believe:

Do I have something wrong?

Thanks @DimitrisMilonopoulos. I haven't seen that one before. It does indeed seem related to padding.
I'll have a look soon.
BTW, are you using Compact mode?
Thanks!

Can reproduce this with compact and without compact :)

Hi @DimitrisMilonopoulos, I've looked into that and unfortunately I couldn't find a fix just yet. The thing I noticed is that if you toggle the replace mode by clicking the green button on the left it fixes itself.


Consider using dedicated namespace for Akri by default

Is your feature request related to a problem? Please describe.
Following the instructions, Akri by default is deployed into the default namespace. Consider using dedicated namespace for Akri by default.
Describe the solution you'd like
Instructions for helm install could contain the desired target namespace to be used.

Hi @jiria this issue may better live in Akri's documentation repository: https://github.com/project-akri/akri-docs. Can you open this there and then we can link to it from here and close this one out?

That would be a documentation fix to include  --create-namespace --namespace akri to our helm install commands.

An example of how prometheus does this: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#redeploy-with-new-name-downtime
We'd need to add similar notes about presumed namespace usage


Regression: switching maps after veto makes matchStarted=true

Caught on 0.8.3 yesterday. This was not the case on update 0.8.1, so I consider it a regression.
Upon completing the veto and changing the map, we go to a warmup where every player should ready up, but plugin decides that matchStarted==true and stops waiting for players to ready up, stops chat messaging and ignores .ready, making match unstartable. Pausing or stopping does not work and leaves the warmup state active.
Without the veto everything is ok.
Seems like it might be an auto-side-selection trouble, if selecting a side makes the player ready.

Log looks like this: [MatchZy] [!ready command] Sent by: 1 readyAvailable: True matchStarted: True
I believe these two variables should always be the opposite of each other :D

Hey @milkywayfarer,
Thanks for the observations and letting me know, helps a lot! I had added Utility.cs:696 (matchStarted = true) in the newer update because the flag was not getting initialised on knife round, but I think thats the wrong place to do it since it messes up with the veto.
I'll put it in the knife round function and test it.

Hey @milkywayfarer,
I've pushed an update which should fix it, can you try with the 0.8.4 version?

Hey thanks for the update, we will get that tested tomorrow on 2 servers

Hey @shobhit-pathak,
We tested the latest version (0.8.5) on 3 servers and everything went as expected, aside some instances where creating a wingman match does not make the gamemode wingman, but it's not related.
Thanks!


The Gatherling scraper hardcodes a start time of 7pm Eastern but Penny Dreadful Sundays start at 1pm Eastern.

This information is just not in the page which is why I hardcoded it in the first place. In fact it's not anywhere on the gatherling site that I can see.
496c69c is a hack.


gojsonschema does not handle duplicate fields

Hello.
gojsonschema does not handle duplicate fields and only keeps the last occurence.
Can you throw an error when processing duplicate fields?
Thank you.
Here is a sample ("firstName" field occurs twice ( "Petr" and "Ivan") ):
package main
import (
"fmt"
"github.com/xeipuuv/gojsonschema"
)
func main() {
src := {         "firstName" : "Petr",         "firstName" : "Ivan",         "lastName" : "Petrov"     }
schema := `{
    "title": "test",
    "type": "object",
    "properties": {
        "firstName": {
            "type": "string"
        },
        "lastName": {
            "type": "string"
        }
    },
    "required": ["firstName", "lastName"],
    "additionalProperties": false
}`

documentLoader := gojsonschema.NewStringLoader(src)
schemaLoader := gojsonschema.NewStringLoader(schema)
result, err := gojsonschema.Validate(schemaLoader, documentLoader)
if err != nil {
    panic(err.Error())
}

if result.Valid() {
    fmt.Printf("The document is valid\n")
} else {
    fmt.Printf("Errors :\n")
    for _, desc := range result.Errors() {
        fmt.Printf("- %s\n", desc)
    }
}

}

I'm not sure this is really a bug with this library. If you run this:
package main

import(
	"encoding/json"
	"log"
	"strings"
)

func main() {
	var document interface{}

	decoder := json.NewDecoder(strings.NewReader(`{ "firstName" : "Petr", "firstName" : "Ivan", "lastName" : "Petrov" }`))
	decoder.UseNumber()

	err := decoder.Decode(&document)
	if err != nil {
		log.Println(err)
		return
	}

	log.Println(document)
}

The output is map[firstName:Ivan lastName:Petrov]
The Go standard library doesn't retain duplicate properties. I'm not really sure what could be done here to detect this.

According to RFC 7159, Section 4:

When the names within an object are not
unique, the behavior of software that receives such an object is
unpredictable.  Many implementations report the last name/value pair
only.  Other implementations report an error or fail to parse the
object, and some implementations report all of the name/value pairs,
including duplicates.

So the Go standard library technically conforms to the JSON specification with its current behavior.  You would have to separately parse out the property names and check for duplicates in order to catch the error.

IIRC it is valid in JSON to have duplicate keys and only the last one is considered when interpreting the document.
F.e. https://stackoverflow.com/questions/244777/can-comments-be-used-in-json/18018493#18018493

@stevenroose valid, yes.  Dependable?  Interoperable?  No.  Which is what that big "WARNING" at the top of that linked stackoverflow thread is about.  You can only use that if you have absolute control over what implementations consume your JSON, including the version of the implementation.  A conforming JSON parser could change behavior in this scenario and still remain in conformance both before and after.


Doesn't allow video to pause when using Plasma Integration plugin

Linux Distro: Kubuntu 20.04 LTS
Kernel: Linux 5.4.0 LTS
DE: KDE 5.18.5 LTS
Browser: Firefox 81.0
I've been experiencing issues when using Plasma Integration plugin and YouTube NonStop plugin in conjunction. Plasma Integration plugin works fine for every other function other than pausing YouTube videos. It pauses for a moment and then continues playback.
Steps to reproduce:

Install Plasma Integration plugin and YouTube NonStop. Order doesn't affect outcome.
Pair another device to KDE Connect and allow media control.
Play a YouTube video and press the pause button.

Thank you.

"dom.media.mediasession.enabled" in about:config wasn't set to true. Excuse me.


Why are ChatRequestTurn and ChatResponseTurn classes and not interfaces.

Testing #213514
Their constructor is private so a user can't create any instances anyways.

In the API you need to be able to distinguish them, and I think that using instanceof is nicer than differentiating them with a string property or something like that. We also do this kind of thing in other parts of the API.


Foyer not working when installed using conda but works with mamba

Bug summary
I'm not sure if this ultimately stems from the foyer or gmso conda packages.
Installing foyer from conda-forge has different behavior depending if you use conda or mamba
It looks like you get different versions of pydantic depending on which method you use.
When installing with conda and trying to import foyer in python I get the following error:
>>> import foyer
Warning: importing 'simtk.openmm' is deprecated.  Import 'openmm' instead.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chris/miniconda3/envs/foyer-conda/lib/python3.10/site-packages/foyer/__init__.py", line 2, in <module>
    from foyer.forcefield import Forcefield
  File "/home/chris/miniconda3/envs/foyer-conda/lib/python3.10/site-packages/foyer/forcefield.py", line 37, in <module>
    from foyer.atomtyper import find_atomtypes
  File "/home/chris/miniconda3/envs/foyer-conda/lib/python3.10/site-packages/foyer/atomtyper.py", line 6, in <module>
    from gmso import Topology
  File "/home/chris/miniconda3/envs/foyer-conda/lib/python3.10/site-packages/gmso/__init__.py", line 2, in <module>
    from .core.angle import Angle
  File "/home/chris/miniconda3/envs/foyer-conda/lib/python3.10/site-packages/gmso/core/angle.py", line 4, in <module>
    from pydantic import Field
ImportError: cannot import name 'Field' from 'pydantic' (/home/chris/miniconda3/envs/foyer-conda/lib/python3.10/site-packages/pydantic/__init__.py)

Installing with conda:
conda create -n foyer-conda -c conda-forge foyer -y
conda activate foyer-conda

I get these versions of foyer, gmso and pydantic:
foyer                     0.11.1             pyhd8ed1ab_0    conda-forge
gmso                      0.8.1              pyhd8ed1ab_0    conda-forge
pydantic                  0.18.2                     py_0    conda-forge

Installing with mamba:
mamba create -n foyer-mamba -c conda-forge foyer -y
mamba activate foyer-mamba

I get these versions of foyer, gmso and pydantic:
foyer                     0.11.1             pyhd8ed1ab_0    conda-forge
gmso                      0.8.1              pyhd8ed1ab_0    conda-forge
pydantic                  1.8.2                     py_0    conda-forge

Software versions
mamba 0.24.0
conda 4.13.0
OS: Linux
I'm curious if anyone else can reproduce this. It's happened or myself and 2 other people in our lab.

It's strange that the 3 numbers in the versions are the same; almost like it's a parsing error or something (I don't really know how conda does all this under the hood).
Maybe one solution is to pin the exact version of 1.8.2 rather than using <1.9.0
conda install pydantic=1.8.2 seems to work for me.

Raised an issue with conda


[1750] DNS domains infrastructure

Description
Create basic resources to manage domains: resource group, storage account, DNS zone and front door
How to review
Check s189p01-capt-dom-rg resource group

 Claim: create zone in s189


Squashing migrations

Context
https://github.com/prisma/prisma/issues/5226#issuecomment-790662655
Suggested action
Add as a separate guide here: https://www.prisma.io/docs/guides/database/developing-with-prisma-migrate - "Squashing migrations"

Covered here: https://www.prisma.io/docs/guides/migrate/developing-with-prisma-migrate/squashing-migrations


Ensure language support

Closes #782

@jpic Care to review?

Looks great, maybe you want to cover get_i18n_name with a few tests, ie. fr-US, fr-FR and whatnot, just to prevent further contributors from breaking your stuff, otherwise it's good for merge :+1:
Keep up the gr8 work :1st_place_medal:

@jpic Good call on the tests...found one issue which has been resolved. Did I put my tests in the right place? I fixed the FAILs caused by my change, but there were a number of ERRORs reported when running test_project/manage.py test.

Looks good to me, good job fixing those tests ! so, any last minute objection or can we merge now ?

I'm good with merging...threw my last second fix in to get a green Travis 👍

Thanks for your help 😄

Thank YOU for your awesome contribution, from what I've read you've made history for many users :medal_sports:
As contributor your are invited to spend some time in our hackerspace in the country side south west of France, where they make fine cognac, pineau, wine and cheese.
Have a fantastic weekend :+1:

Could a new release be made that has this fix, please? It appears that the most recent release was in April.

Released in 3.3.5

Thank you!


Jenkins runner support

Chainloop has support for CI/CD runners, this task is about adding Jenkins (and JenkinsX) to the list of supported ones.
On the technical side it meens implementing this interface https://github.com/chainloop-dev/chainloop/blob/053637d91d5aa728d50ec3740f246fdca8120b87/internal/attestation/crafter/runner.go#L27-L37
which will require you to inspect and figure out 1) how to know that you are in that runner, 2) what env variables you care about for the tool to auto-discover and 3) how to create a link to the workflow.
Additionally you'll need to update the contract definition to support this new runner type
https://github.com/chainloop-dev/chainloop/blob/053637d91d5aa728d50ec3740f246fdca8120b87/app/controlplane/api/workflowcontract/v1/crafting_schema.proto#L36-L40

Looking after this.

This is interesting:
During the installation of jenkins (in this particular case, I'm using a docker container), the following setup step happens through the guided procedure:

I will need to double-check it but as the wizard says this configuration is required, most likely the RunURI() method can be implemented using these values. I am assuming this could be retrieved from the job's node.

Correct, I believe that URL matches JENKINS_URL env var
https://e.printstacktrace.blog/jenkins-pipeline-environment-variables-the-definitive-guide/

To make sure, I am creating an alias to <IP_ADDRESS> that is different than localhost. I will use such alias to configure jenkins, to see if it appears in the job's node. That will be the ultimate confirmation.

Just some notes, mostly for myself.
If we SSH into a jenkins agent node, we can see that the info available at system level is not much:

The env variable AGENT_WORKDIR should be guaranteed to be present.
If I launch a job on the node, the set of running processes and the set of environment variables do not change at all. This means that the Groovy process is ran inside agent.jar, which by the way should also be guaranteed to be present on the node, and should have that particular name. Specifically, its pathname should end with the suffix jenkins/agent.jar.
Instead, if we are the jenkins job, we have a lot more info at our disposal. The set of environment variables is the following:
JENKINS_HOME=/var/jenkins_home
CI=true
RUN_CHANGES_DISPLAY_URL=http://jenkins-master:8080/job/simple-pipeline/9/display/redirect?page=changes
HOSTNAME=a5bcfe8f5f10
NODE_LABELS=root-inbound-agent docker
HUDSON_URL=http://jenkins-master:8080/
HOME=/home/jenkins
BUILD_URL=http://jenkins-master:8080/job/simple-pipeline/9/
HUDSON_COOKIE=f0961d75-4083-4ace-9ea6-4f7ea5a48716
JENKINS_SERVER_COOKIE=durable-eda971e81cf14777d9b1fed11303b0dda56bb649f09a6e05da324f4698fa9cc5
WORKSPACE=/home/jenkins/agent/workspace/simple-pipeline
NODE_NAME=root-inbound-agent
RUN_ARTIFACTS_DISPLAY_URL=http://jenkins-master:8080/job/simple-pipeline/9/display/redirect?page=artifacts
STAGE_NAME=Inspect environment
EXECUTOR_NUMBER=0
user=jenkins
RUN_TESTS_DISPLAY_URL=http://jenkins-master:8080/job/simple-pipeline/9/display/redirect?page=tests
BUILD_DISPLAY_NAME=#9
HUDSON_HOME=/var/jenkins_home
AGENT_WORKDIR=/home/jenkins/agent
JOB_BASE_NAME=simple-pipeline
PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
BUILD_ID=9
BUILD_TAG=jenkins-simple-pipeline-9
LANG=C.UTF-8
JENKINS_URL=http://jenkins-master:8080/
JOB_URL=http://jenkins-master:8080/job/simple-pipeline/
BUILD_NUMBER=9
JENKINS_NODE_COOKIE=b6209db8-1376-4931-89e7-2b9a46862332
RUN_DISPLAY_URL=http://jenkins-master:8080/job/simple-pipeline/9/display/redirect
HUDSON_SERVER_COOKIE=a4d8db8ebe73e068
JOB_DISPLAY_URL=http://jenkins-master:8080/job/simple-pipeline/display/redirect
JOB_NAME=simple-pipeline
JAVA_HOME=/opt/java/openjdk
PWD=/home/jenkins/agent/workspace/simple-pipeline
WORKSPACE_TMP=/home/jenkins/agent/workspace/simple-pipeline@tmp
TZ=Etc/UTC

The set of running processes is much richer than before. I am not pasting it here because the environemtn variables are going to be more than enough to guarantee that we can detect a jenkins node.

In my opinion, to detect that we are in a jenkins job, we could check the presence of all of the following environment variables:

JENKINS_HOME
BUILD_URL
NODE_NAME
AGENT_WORKDIR

As optional, we could also detect that the process agent.jar is running. I don't think it's necessary.

Thanks for digging into this and sharing your findings.
Your suggestion makes sense to me, although let me share some thoughts in case you think it makes sense.
Detection via CheckEnv()
Note that this feature is not a security enforcement but more like a duck-typing kind of detection. So, to me the minimum we could add, the better.
So far, for other runner types, my choice was based on

a) what's something that could unequivocally identify this runner type and
b) what other env vars do we need to generate the JOB URL

So it seems that in your case, we'd have enough with making sure that both JENKINS_HOME + BUILD_URL are present, don't you think?
Env Vars listing ListEnvVars()
This method is used to automatically inject env vars into the attestation. I'd recommend to check what variables we use in other runners to support a similar context.
Long story short, what we try to add is some representation of:

Who triggered the job (in your case maybe the user env var)
Some information about the commit (I can't see that in your vars, it might not exist since it was not triggered automatically by a change?)
Some information about the agent where the job is running (you have AGENT_WORKDIR and maybe NODE_NAME)

This is for example what we do for GitHub actions.
https://github.com/chainloop-dev/chainloop/blob/053637d91d5aa728d50ec3740f246fdca8120b87/internal/attestation/crafter/runners/githubaction.go#L42-L52
In any case, feel free to create a PR with your suggestions, there is not right or wrong answer here and we can iterate later on!
THank you!!

Yep, I think you're very right. I was going to use those 2 vars only :P
I didn't check Jenkins X yet, though. Is it a problem if I discover that Jenkins X instantiate the same 2 variables?

I didn't check Jenkins X yet, though. Is it a problem if I discover that Jenkins X instantiates the same two variables?

I'm not 100% sure, but I think Jenkins and JenkinsX are different enough to be implemented as different runner types. So, ideally, a Jenkins X run should not impersonate a regular Jenkins run.
That said, and after looking at this JenkinsX reference doc, your proposal seems to work as expected.

From what I read yesterday about Jenkins X, I share the feeling that they should have different runners.
I still don't know very well what Jenkins X is, so I was planning to do cover it afterwards in another PR, unless you wish otherwise.

From what I read yesterday about Jenkins X, I share the feeling that they should have different runners. I still don't know very well what Jenkins X is, so I was planning to do cover it afterwards in another PR, unless you wish otherwise.

Makes sense to me!
Thanks!


Clarify which "Insights" this doc is for

[Enter feedback here]
On this page does "Insights" refer to Playfab Insights or Azure Application Insights?
Need to clarify as Google search has this as the top results when you search for Kusto Explorer Application Insights.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0172001c-f008-dd94-6315-669e92cb8a8a
Version Independent ID: 4963782a-12ce-c4dd-d7e8-7f0d3794963c
Content: Connecting Kusto Explorer to Insights - PlayFab
Content Source: playfab-docs/features/Insights/connectivity/connecting-kusto-explorer-to-insights.md
Product: playfab
GitHub Login: @natashaorie
Microsoft Alias: norie

@LDS2020 thos documentation is about PlayFab Insights.


Failing to get afj running in Expo for ios

I followed this tutorial & I am on a macbook pro M1 with Ventura 13.2:
https://www.youtube.com/watch?v=Ho9Db0PSaGM
I get stuck at the end when running:
expo run:ios

The error I get in the console:
The following build commands failed:
SwiftEmitModule normal arm64 Emitting\ module\ for\ indy_sdk_react_native (in target 'indy-sdk-react-native' from project 'Pods')

In the xcode build log I found:
<unknown>:0: error: unable to load standard library for target 'arm64-apple-ios10.0'
<unknown>:0: error: module compiled with Swift 5.5.2 cannot be imported by the Swift 5.7.2 compiler: /Users/tim/Projects/iWize/timboektoe/aries-poc/react-native/wallet/ios/Pods/Indy/IndySdk/Indy.framework/Modules/Indy.swiftmodule/arm64.swiftmodule

Googling this error I stumbled upon:
https://stackoverflow.com/questions/58654714/module-compiled-with-swift-5-1-cannot-be-imported-by-the-swift-5-1-2-compiler
Setting BUILD_LIBRARY_FOR_DISTRIBUTION = YES; might be a solution.

Did you enable bitcode as stated in the docs? https://github.com/animo/indy-sdk-expo-plugin#getting-started

Yes, I did. I noticed however that my IDE stated that this property was not allowed here.

See my comment in the other issue re Expo and react native version used

Switched to expo 47, but still:
** BUILD FAILED **

The following build commands failed:
        SwiftCompile normal arm64 Compiling\ IndySdk.swift /Users/tim/Projects/iWize/timboektoe/aries-poc/react-native/wallet/node_modules/indy-sdk-react-native/ios/IndySdk.swift (in target 'indy-sdk-react-native' from project 'Pods')
        CompileSwift normal arm64 /Users/tim/Projects/iWize/timboektoe/aries-poc/react-native/wallet/node_modules/indy-sdk-react-native/ios/IndySdk.swift (in target 'indy-sdk-react-native' from project 'Pods')
(2 failures)

<unknown>:0: error: unable to load standard library for target 'arm64-apple-ios10.0'
<unknown>:0: error: module compiled with Swift 5.5.2 cannot be imported by the Swift 5.7.2 compiler: /Users/tim/Projects/iWize/timboektoe/aries-poc/react-native/wallet/ios/Pods/Indy/IndySdk/Indy.framework/Modules/Indy.swiftmodule/arm64.swiftmodule

Which version of xcode are you using?

14.2

Closing as a lot has changed. If the issue still persists with latest 0.5.0 alphla, node 18/20 and 0.2.0 of the shared components, please open a new issue. Thanks!


Review existing spreadsheet and subcategory of conditions

I don't know that this means, closing.  Create a new more specific issues, if needed.


Sort of add working active classes

#14
Added working active menus for top-nav layout, but the sidebar layout for some reason does nothing.
Note that this is extremely wonky and doesn't account for dynamically generated URL. Only the URLs specified in the adminlte.php in the config folder are going to have an active class.
If all URLs were to be converted to routes, it would be a bit easier. But then again, you can just add this package for that.

Thanks! It seems to work better than you're explaining: for me it works in the sidebar as well, and also dynamically generated URLs work. The code is not perfect, but I think it is ok for now.


Less concise simplification of markers in uv 0.2.35

Noticed in https://github.com/python-trio/trio/pull/3068, I haven't tried to get a smaller case. The command is uv pip compile --universal --python-version=3.8 test-requirements.in -o test-requirements.txt. Platform is Ubuntu, uv is version 0.3.0 but we didn't run uv --version.
uv 0.3.0 tries to make this change:
-colorama==0.4.6 ; sys_platform == 'win32' or (implementation_name == 'cpython' and platform_system == 'Windows')
+colorama==0.4.6 ; (implementation_name != 'cpython' and sys_platform == 'win32') or (platform_system != 'Windows' and sys_platform == 'win32') or (implementation_name == 'cpython' and platform_system == 'Windows')

This might mean https://github.com/astral-sh/uv/pull/5898 doesn't simplify in this case as well. Maybe it's missing some necessary case?
Why is (implementation_name != 'cpython' and sys_platform == 'win32') or (platform_system != 'Windows' and sys_platform == 'win32') or (implementation_name == 'cpython' and platform_system == 'Windows') equivalent? (just to show the previous result wasn't just a bug and since it was a bit confusing to me)

markers
why is this equivalent

(implementation_name != 'cpython' and sys_platform == 'win32') or (platform_system != 'Windows' and sys_platform == 'win32') or (implementation_name == 'cpython' and platform_system == 'Windows')
its what we start with

(sys_platform == 'win32' and (implementation_name != 'cpython' or platform_system != 'Windows')) or (implementation_name == 'cpython' and platform_system == 'Windows')
distribution

((implementation_name == 'cpython' and platform_system == 'Windows') or (sys_platform == 'win32')) and ((implementation_name == 'cpython' and platform_system == 'Windows') or (implementation_name != 'cpython' or platform_system != 'Windows'))
distribution

((implementation_name == 'cpython' and platform_system == 'Windows') or (sys_platform == 'win32')) and ((implementation_name == 'cpython' and platform_system == 'Windows') or NOT (implementation_name == 'cpython' and platform_system == 'Windows'))
demorgan's law

((implementation_name == 'cpython' and platform_system == 'Windows') or (sys_platform == 'win32')) and TRUE
negation

(implementation_name == 'cpython' and platform_system == 'Windows') or (sys_platform == 'win32')
identity

Minimal reproduction:
x.in:
pytest >= 5.0
black; implementation_name == "cpython"

Then run uv pip compile x.in --no-strip-markers. I can reproduce this in 0.2.35 and not in 0.2.34. uv --version: uv 0.2.35 (e097f948c 2024-08-09), reproducing on Windows.

I believe this is because we simplify to DNF as a normalized representation. In some cases, though, CNF would be more concise. I think this can be merged into https://github.com/astral-sh/uv/issues/5992, but let me know if you disagree.

I don't think so, because the simpler version only has or at the top level. But maybe that's preventing one of the simplification steps?

Here's a comment from our code for DNF simplification:
/// Simplifies a DNF expression.
///
/// A decision diagram is canonical, but only for a given variable order. Depending on the
/// pre-defined order, the DNF expression produced by a decision tree can still be further
/// simplified.
///
/// For example, the decision diagram for the expression `A or B` will be represented as
/// `A or (not A and B)` or `B or (not B and A)`, depending on the variable order. In both
/// cases, the negation in the second clause is redundant.
///
/// Completely simplifying a DNF expression is NP-hard and amounts to the set cover problem.
/// Additionally, marker expressions can contain complex expressions involving version ranges
/// that are not trivial to simplify. Instead, we choose to simplify at the boolean variable
/// level without any truth table expansion. Combined with the normalization applied by decision
/// trees, this seems to be sufficient in practice.
///
/// Note: This function has quadratic time complexity. However, it is not applied on every marker
/// operation, only to user facing output, which are typically very simple.

The main issue here is that DNF on its own isn't actually unique. And full simplification might not be feasible. But I think this might require more investigation. But I think this can be safely folded into #5992, which is, I think, the same problem that is reported here: the marker is correct but not as simple as it could be.
It is really important though that we have something approach a canonical representation for markers, otherwise the lock file might not be stable. This may come at the cost of concise markers.


Selecting ADS on CCC does not produce results

https://www.ospc.org/ccc/384/

cc @PeterDSteinberg @jdebacker

This is a problem on both the test app and on ospc.org.

@MattHJensen and @jdebacker - Brittain suggests this can be closed, please confirm.

Closing.


kube-aws: improve documentation of externalDNSName option

#hackday
Currently:
This DNS name is assumed to be routable from outside the cluster to the Kubernetes API.
It is automatically added as a Subject Alternative Names (SANs) to the kube-apiserver TLS certificate.

Couple of questions

Do I have to provision this before running kube-aws up?
For development can I use a DNS name without actually provisioning a DNS name correctly?
What IP address should this DNS name point to?

+1 came here after googling and have no idea what I should be doing with this

All it does is to make sure the TLS cert cover this "DNS" or "domain" name so later when you try to connect you can get a secure connection.
For testing purpose you don't need to get a real domain name at all, e.g. you can give it whatever name you like, I normally give it a "kube1". After my kube-aws up run finish and the public IP is printed, I just do"
sudo vi /etc/hosts

and add/edit a line:
<IP_ADDRESS>    kube1

Then
export KUBECONFIG=/home/dong/aws/kube-aws/clusters/kube1/kubeconfig

Now you are ready to fire your kubectl cmds.

related: https://github.com/coreos/coreos-kubernetes/issues/83

Closing as duplicate of https://github.com/coreos/coreos-kubernetes/issues/83


swap getopt with argparse

Hi
argparse is more powerful with less code to handle command line arguments processing.
Please review and comment.
[me@salt01 example]$ ../brine.py
usage: brine.py [-h] -b BRINE [-d]
brine.py: error: argument -b/--brine is required
[me@salt01 example]$ ../brine.py -d
usage: brine.py [-h] -b BRINE [-d]
brine.py: error: argument -b/--brine is required
[me@salt01 example]$ ../brine.py -d -b test01.ele
brine file: test01.ele
[me@salt01 example]$ ../brine.py -b test01.ele -d
brine file: test01.ele
[me@salt01 example]$ ls -lrt
total 20
-rw-rw-r-- 1 me me  100 Jul 11 08:01 Brinefile.empty
-rw-rw-r-- 1 me me  384 Jul 11 08:02 test01.role
-rw-rw-r-- 1 me me  640 Jul 11 08:02 test01.ele
drwxrwxr-x 2 me me    6 Jul 11 08:08 files
-rw-rw-r-- 1 me me 1240 Jul 11 08:09 init.sls
-rw-rw-r-- 1 me me  528 Jul 11 08:09 README.md
[me@salt01 example]$

Here is output of -h argument.
[me@salt01 brine]$ ./brine.py  -h
usage: brine.py [-h] -b BRINE [-d]

Brine 1.0 is a Salt Formula initializer. https://github.com/openx/brine

optional arguments:
  -h, --help            show this help message and exit
  -b BRINE, --brine BRINE
                        Ex: -b test01.ele
  -d, --debug           log debug info into brine.log
[me@salt01 brine]$

@tjyang Looks like there are conflict that are keeping us from merging these changes. Can you rebase from upstream, apply your changes and resubmit the PR?


Create home page component - Vanilla.js

Create home page component - Vanilla
🚀 Goal
Create a home page component in Vanilla that serves as the landing page for our Decentralized Web App (DWA) starter, providing different views based on the user's connection status.
🤔 Background
We need to implement the home page for our Vanilla.js DWA starter, mirroring the functionality of the existing React implementation. This page should display a welcome message and a connect button when the user is not connected, and show a TodoList component (or placeholder) when connected.

This is part of our larger project to create a Vanilla.js DWA starter. See our main issue here for the full context and list of all related tasks.

Important: For reference, please see the DWA React Vite starter app. While the implementation details will differ for Vanilla, this example provides a good overview of the structure and functionalities of a DWA.

🔑 Tasks and Acceptance Criteria

[ ] Create a HomePage.vanilla component
[ ] Implement conditional rendering based on the user's connection status:

[ ] When not connected, display:

[ ] A title "DWA Starter"
[ ] An image (use the same workplace.svg or an equivalent)
[ ] A button labeled "Connect to your DWA" that initiates the Web5 connection process
[ ] A message below the button saying "Connect your DWA to get started"

[ ] When connected, display:

[ ] If a TodoList component exists, use it here
[ ] If no TodoList component exists yet, create a simple placeholder component that says "Todo List Coming Soon!"

[ ] Ensure the component uses the Web5 connection status correctly
[ ] Implement the connection functionality when the button is clicked
[ ] Style the component using Tailwind CSS to match the existing design
[ ] Ensure the layout is responsive and looks good on various screen sizes

🌟 Resources

Web5 Documentation
Vanilla JS Documentation
Tailwind CSS Documentation
DWA React Vite Starter App (for reference)

Getting Started

Comment ".take" on this issue to get assigned
Fork the repository and create a new branch for this task
Follow the tasks outlined above
Submit a pull request with your changes
Respond to any feedback during the review process

Questions?
If you have any questions or need clarification, please comment on this issue or join our Discord community.
Happy coding! 🎉

.take

can you describe what kind of image should i add

@29deepanshutyagi hiii! If you run this project locally, you will be able to see what it should look like https://github.com/TBD54566975/tbd-examples/tree/main/javascript/dwa-starter-react-vite
But it should look like this

Remember this is a JavaScript version of the React project.