[RFC]: Adding support for Geospatial models

Motivation.
Modern models are now not only targeting the generation of text but also the generation of images from text or image input as well. This RFC wants to open the stage towards supporting models that do not only generate text but also images either as a single modality or even supporting multi-modal output. One example of great interest to us is a set of models developed in co-operation with NASA for earth observation (https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M) working on satellite images can be fine-tuned for several tasks including floods forecast, crop classification etc.
This example model, works on images of a fixed size in input and generates an image of the same size in output. In the specific, input images in the geotiff format are split in patches of dimensions 224×224, each patch is passed through the model for inference that generates a tensor of the same size as the input. This is similar in a way got an autoregressive process, with the difference that at every iteration the data passed to the model is different and there is no relationship when inferencing subsequent patches. All the output patches are then “re-assembled” into a geotiff image.
The goal of this RFC is that of creating enablement for non text output and demonstrate this with the above mentioned model.
Why support models in vLLM that do not generate text? Because consolidating towards a single serving platform simplifies the software stack of those dealing with multiple types of models. Also, with time, models not targeting text might benefit from optimizations introduced by the vLLM community. Similarly to what has been happening for Transformer based causal models.
Proposed Change.
I propose a two phase approach. In the first phase, integrate the model as a PoolingModel and pre/post-process input output data outside of vLLM. In the second phase, a proper integration of the models is performed, taking also care of processing the input image and generating the output one.
Phase 1: Basic enablement of Geospatial model in vLLM
Pre/post processing of the input image is done outside of vLLM. The input image is broken down into patches (generic tensor), all patches are fed into vLLM. Output tensors are collected and post processing would re-create the output image.
For this phase we could piggyback on the support available for pooling models (thanks @dar for the suggestion) where the hidden states of the model are returned in output.
Changes for phase 1:
Step 1
Extend the output type for pooling models, currently only targeting embeddings, to also support a generic output type. This output would be then post processed outside of vLLM.
@dataclass
class PoolingOutput:
    """The output data of one pooling output of a request.

    Args:
        outputs: This can be either a list of floats (embedding vector), or a generic list of tensors defined by the model. The embedding vector, returned in case of embedding models, is a list of floats whose length depends on the model as listed in the embedding guide.
    """
    outputs: Union[List[float], List[torch.Tensor]]

Step 2
Also, right now, for Pooling models the only two possible methods to execute would be encode or score. Would it make sense to define a third one like transform? This would just be for the sake of not using encode. Also, would it make sense to create a new entrypoint class in addition to LLM? Something Like VisionModel or similar? This is again for sake of completeness since this is not a language model.
Step 3
Exploit the batching capabilites of vLLM and present all the images patches to the vLLM entrypoint as a list of generic tensors. Similar to what is done now when presenting multiple prompts at a time.
Phase 2: Optimized integration of the model
Embed pre/post processing of images into vLLM and handle the recursive pattern for processing bigger images in vLLM.
(This phase might need to be updated/changed depending on the outcome of Phase 1)
Step 1
Integrate processing of the input with the already available multimodal input support. Here among the things to be considered is that an input image could be presented encoded as a string instead of being stored in an file.
Step 2
Introduce the possibility of “installing“ an output processor, that generates images of the required format. In the same spirit of what is done for input processors.
@INPUT_REGISTRY.register_input_processor()

The idea would be to create an output registry and enable models to register an output processor so that all the output generated for a sequence can be converted into the proper image format for the specific model.
Step 3
Create a new output class that allows the output to be presented in the form of an image. We could call it ImageOutput and ImageRequestOutput. Users would be able to either post-process the model output and return a string containing the generated file or, return the raw image output for post-processing outside of vLLM
class ImageOutput:
    image_out: Union[str, torch.Tensor]

class ImageRequestOutput:
    def __init__(self, request_id: str, outputs: "ImageOutput",
                 finished: bool):
        self.request_id = request_id
        self.finished = finished
        self.outputs = outputs

Step 4
Handle recursive processing of image patches within vLLM. Each image is fed to vLLM, pre processed and split in patches. All patches are processed and all the output patches are processed by the output processor. Could we re-use some of the logic used for handling autoregressive queries? In this case we would know already how many times the model inference should be executed (the number of image patches) and no need to append the output of an iteration to the input of the next, we just feed the next patch and so on.
The output of the request in this case will still be of type ImageRequestOutput with the image_data field actually optional and the image_path populated with the path to the image generated during post processing
Feedback Period.
2 weeks
CC List.
@njhill @ywang96 @DarkLight1337 @robertgshaw2-neuralmagic
Any Other Things.
No response
Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Hey @christian-pinto! Thank you for the interest and the putting the effort this RFC!
First of all, I would like to clarify that when it comes to model support, IMHO it is less about if we're generating text versus other modalities, but whether or not the underlying backbone of the architecture is autoregressive decoder-only transformer with causal attention. This was the main motivation when we first added multimodality inputs & models a while back, and what every multimodal model on vLLM belongs to today (with the only exception being multimodal llama 3.2 which we had a lot of challenges from supporting it). Therefore, I do not expect vLLM to provide significant performance improvement for models that don't fall into this category.
Now moving on to infra support for generating non-text modalities. This was something I also seriously thought about when I ported Chameleon to vLLM (an autoregressive VLM that's able to generate not only text tokens but also image tokens to be decoded to images). I do see the value of consolidating use cases to a single platform, thus I'm not particularly against opening up the support for image generation. However, there are a few things I'm concerned about and we should think out loud about before we proceed:

Can you share some pseudocode on what user interface will look like? Do you plan to add support for both online & offline inference?
We should keep the codepath for this kind of models separate from the main LLM support as much as possible (similarly to what we did with embedding models) so that the two don't intervene each other. This also means when we design optimizations for the main LLM support, we may not necessarily take other types of models into account.
I think it's also of vLLM's interest to have models and/or features added if they can be also maintained over time. A few great examples are LoRA support from @jeejeelee, tool calling from @K-Mistele, etc. I'm curious if you have plans to maintain multimodal output infra going forward as well.

Hope this make sense and let me know what you think!

Hi @ywang96, many thanks for your comments.
I do understand that most of the optimizations that are in vLLM are not going to provide any immediate benefit for models that are not  autoregressive with causal attention. This RFC is in fact coming from the need for supporting multiple types of models with the same software stack. My hope is that by adding support for "different" types of models at some point we might start seeing optimizations to land also outside the auto-regressive decoder-only umbrella.
To reply to your points one by one:

Can you share some pseudocode on what user interface will look like? Do you plan to add support for both online & offline inference?

I would start with the offline inference, to get the main pieces in place and then extend of online as well.
as per the user interface at the moment I have in mind something like the below (targeting the model mentioned in this RFC)

model = LLM(model="./test_model", skip_tokenizer_init=True) # Perhaps a different entry point could be defined since these are not LLMs anymore
input_image = read_image("/path/to/the/image")

image_patches: List[torch.Tensor] = process_image(input_image)

model_outputs = model.transform(image_patches)

output_patches: List[List[torch.Tensor]] = []
for output in model_outputs:
    output_patches.append(output.outputs.tensors) # this would be an extension to the PoolingOutput class (different fromthe one I propose on top)

process_output(output_patches, "/path/to/output/image")

Worth noting in the above snippet are:

model_outputs = model.transform(image_patches): this comes from my proposal to add a method for Pooling models that avoids using sample or score and that takes a generic tensor in input.
output_patches.append(output.outputs.tensors): in my original comment I was proposing extending the PoolingOutput differently, but it could actually be extended as follows. WE can find a better name other than just tensors.

class PoolingOutput:
    """The output data of one pooling output of a request.

    Args:
        embedding: The embedding vector, which is a list of floats. The
        length of vector depends on the model as listed in the embedding guide.
    """
    embedding: Optional[List[float]]
    tensors: Optional [List[torch.Tensor]]

Basically, for a first implementation I would heavily re-use whatever is there for pooling models and avoid introducing heavy changes to the current structure, that would also take care of "disabling" the autoregression (see embedding models).
In a second phase I would propose to create  a proper abstraction for these models, where there is a dedicated class for image output and users can register an output image processor that converts the output of the model back into an image.
basically similar user interface but developers will have to decorate their model with something like.
@MULTIMODAL_REGISTRY.register_output_image_processor

Or perhaps define an OUTPUT_REGISTRY.
so in this case the user would interact with the model differently as the processing of input and output would be performed in vLLM using the input and output processors registered for the model.

We should keep the codepath for this kind of models separate from the main LLM support as much as possible (similarly to what we did with embedding models) so that the two don't intervene each other. This also means when we design optimizations for the main LLM support, we may not necessarily take other types of models into account.

Absolutely, totally understandable and I agree.

I think it's also of vLLM's interest to have models and/or features added if they can be also maintained over time. A few great examples are LoRA support from @jeejeelee, tool calling from @K-Mistele, etc. I'm curious if you have plans to maintain multimodal output infra going forward as well.

I would be able to maintain this part for the foreseeable future.

As a first step, I have created #11129 to enable different types of pooling outputs. PTAL!

As a first step, I have created #11129 to enable different types of pooling outputs. PTAL!

Thanks @DarkLight1337, that is exactly along the lines of what I had in mind for modifying the output of pooling models. I will submit a PR with the new model in a couple of weeks right after the Christmas vacation break.


Bump to new version

This for 0.5.4 in advance - as I forgot to do 0.5.3 after the last release.

Didn't we start bumping the version last release and it broke something so we held back?

I believe it was something with the auto downloader, because 0.5.4 isn't out yet.  I am okay with this ticket sitting until we do the release.  I just think it's good we create after we do the prior release so we don't forget about it.


Fixed picker scroll glitch

Details
A detailed explanation of the issue & this PR's solution, can be found here https://github.com/Expensify/Expensify.cash/issues/3023#issuecomment-854886532
Fixed Issues

Fixes #3023
Tests

Open Emoji Picker.
Scrolled to the end of the emojis list.
Scrolled up & own many times to see if the glitch exists anymore.
Used Arrow keys to navigate the emojis.

QA Steps

Open E.cash.

Open any chat.

Open the Emoji picker from the composer.

Scroll to the end of the Emoji List.

when the end is reached, there should not be a scroll glitch like
https://user-images.githubusercontent.com/44479856/118901476-606f6e80-b8e1-11eb-8936-22c29239237c.mp4

Tested On

[x] Web
[x] Mobile Web (only Affected Platform)
[x] Desktop
[x] iOS
[x] Android

Screenshots

Web

Mobile Web

https://user-images.githubusercontent.com/24370807/120854648-13ef8880-c59b-11eb-91ed-62967ebef444.mp4
Desktop

iOS

Android

🚀 Deployed to staging in version: 1.0.63-1🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production in version: 1.0.64-0🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


QR code scanner does not work on iPhone

If text in qr code contains JSON.  Event qrTextReceived does not work.

Hi @dvadert Can you attach an example? I guess that this is not working on any phone no just Iphones!

It does work on Xiaomi at least, and I guess on all android devices, but not working on Iphones seems like an Iphone issue (or security feature?). Actually almost had heart attack about that, cause I tested my app on andoid all the way up to couple days before release. Just switched from JSON to text string with separators.


Release Management APIs

Hi @adamdriscoll - great module and looks to be just what I've been looking for! :grin:
I'm wondering if you're planning on implementing the new release management APIs?
https://www.visualstudio.com/en-us/docs/integrate/api/rm/overview
I'm happy to do this and submit a PR to add the support if you're interested?
Cheers!

Closing this one as I included these in PR #34


Add sample project workflow

Describe the problem this PR is solving
Adds a sample project workflow to allow extension uses to clone a Couchbase sample project and open / add it to their workspace. The extension asks for a project name and local folder to put the example then downloads the example. Once the example has been downloaded, the extension asks if they would like to open the example or add to the current workspace.

Closes #139

Short description of the changes

Adds gitly as extension dependency
Add and wire up new action in package.json & extension.js
Add webview that shows a list of sample projects grouped by language
Add workflow to the click event of selecting one of the examples

TODO:

[ ] Improve styling of sample webview

@MikeGoldsmith
Bug Report: Uncaught Error "An instance of the VS Code API has already been acquired" when Clicking on Project Links after Cancelling while it ask for Project Name
To reproduce this issue, please follow these steps:

Click on "New Project from Sample Template..." in the interface.
Select one of the available links.
When prompted to enter a project name, cancel the operation by pressing the Esc key.
Observe the warning message "Project name is required," which is the expected behaviour.
Attempt to click on any other project within the same window.
At this point, you will encounter the "Uncaught Error: An instance of the VS Code API has already been acquired" error.

Please investigate this issue and take appropriate action to resolve it. Let me know if you require any additional information or steps to reproduce the bug.

Abnormal behaviour if I choose already user folder name for my new project
Here's a step-by-step description of the issue:

When creating a new project, let's say "test1" with Python as the chosen language.
Subsequently, attempting to create another project in the same folder, this time with Java as the language and again using the name "test1".
The result is that the files from the new Java project overlap with the existing Python project in the project structure, leading to an unintended combination of both Python and Java files within the same folder.
To provide a more desirable user experience, it would be beneficial if an error message could be displayed, notifying the user that a project with the name "test1" already exists in the selected folder. This would prompt the user to either choose a different name for the project or select an alternative folder to avoid file overlap.

@AayushTyagi1 I've updated the workflow to check if the project uri already exists, and show an error it does. This prevents a user from re-using an existing directory for the sample to be cloned into.


Add react and webpack to the project

I would really like to add more structure to the project. To achieve this I want to add React and TypeScript to this project with webpack for the build proces.
See example: https://www.electronforge.io/guides/framework-integration/react-with-typescript

Fixed in merge React refactor #7


[BC API 2.0 / POST customerPaymentJournals] Remove read-only properties

Removed both balancingAccountNumber and lastModifiedDateTime from the request body, as they are read-only.

Learn Build status updates of commit e3bbc1d:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

dev-itpro/api-reference/v2.0/api/dynamics_customerpaymentjournal_create.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


Add an example of reading the learning rate from the optimizer state

This question comes up relatively often, let's put it clearly in the documentation.

I'd like to work on this!

That's great, thanks a lot! The replies in #206 might be helpful for showing the different ways of doing it. Let us know if you have any questions!


what does capture() returns?

it says

Image data from capture() is 12-bit, non-normalized (raw sensor data).

I have a lepton 2.5 dev kit
I would assume it should be the temperature value for each pixel
but when I put a melting ice cube (arround 0 degree C) and a cup of coffee (tested with probe, arround 55 degree  C) in front of the sensor, the minimum in the array is 27415 and maximum value is 32709 with doesn't really make sense to me

The values are actually in centikelvin, which is (K * 100).
So if you have a value of 27415, it means that the temperature value is 274.15 Kelvin, which is 1 Celsius.

Oh thank you so much for this. Never would have thought it's kelvin 🤣


Check script doesn't run format check

What do you mean?

He means continuous-integration/check.sh doesn't do this particular check.

Ah! I forgot check.sh exists.


Clippy Pedantic Acceptable Commits Try Again

I changed the wording of a commit to include clippy::.
https://github.com/uutils/coreutils/pull/6738
https://github.com/uutils/coreutils/pull/6746

You can force push to branches to overture old commits so you don't have to create new PRs. I'll try to look at this tomorrow

@sylvestre @cakebaker How are these GNU tests doing? Are they usual suspects for intermittently failing tests?

@tertsdiepraam: yes, they are the usual suspects for intermittently failing tests :|
@dcampbell24: thanks!

Yes, please use force-push in the future to preserve discussions about a PR :)

Sure, I didn't know that I could do that.


Adding waiHasPopup on widgets extending ActionWidget

This PR adds the waiHasPopup property on all widgets which extend ActionWidget (@aria:Button, @aria:Link, ...).

@fbasso Thank you for your review!


reflections on task 56

Absolutely, here are the steps taken in the form of an algorithm along with instructions for creating the "Reflections on Task 56" meta-task:
Algorithm: Transform GitHub Issue into Meta-Task

Access GitHub Issue Data:

Retrieve the GitHub issue data from the provided URL.
Extract the relevant information such as the issue title, body, and comments.

Map to OpenAPI Task Specification:

Map the issue title and body to the title and description fields in the OpenAPI task specification.

Define Paths and Operations:

Analyze the issue content to identify the appropriate paths, methods, and operation IDs for the API calls.
Map the issue context to the corresponding paths and methods.

Create Request and Response Examples:

Utilize examples from the issue to define request and response bodies for each API call.
Incorporate these examples into the OpenAPI task specification.

Define Schemas for Components:

If necessary, create schemas for any models, structures, or objects referenced in the issue.
Ensure that these schemas are reusable and consistent.

Annotate with Metadata:

Add annotations like tags, external documentation, security schemes, etc., to the OpenAPI specification to enhance clarity and usability.

Translate to API Calls:

Translate each step of the OpenAPI task specification into a series of API calls.
Implement these API calls in a programming language using appropriate HTTP libraries and frameworks.

Instructions: Reflections on Task 56 (Transform GitHub Issue)

Access GitHub Issue:

Open the GitHub issue titled "create azure infrastructure" using the provided URL: https://github.com/meta-introspector/meta-meme/issues/56#issuecomment-1691855068

Extract Issue Details:

Analyze the issue to identify its core details, including the title, body, and comments.

Map to OpenAPI Task:

Use the title and body of the GitHub issue to create the basis for an OpenAPI task specification.
Designate paths, methods, and operation IDs based on the issue context.

Create Request and Response Examples:

Identify any examples provided in the issue for API request and response structures.
Incorporate these examples into the OpenAPI specification for accurate representation.

Define Component Schemas:

If the issue references any models or structures, create reusable schemas for them in the OpenAPI specification.

Annotate with Metadata:

Enhance the OpenAPI specification with annotations like tags, external documentation, and security schemes.

Translate to API Calls:

Translate each step of the OpenAPI specification into a series of practical API calls.
Implement these API calls in a programming language using appropriate libraries and frameworks.

Document Reflections:

As you go through the transformation process, document your reflections, insights, and challenges.
Consider how the GitHub issue translates into a functional API specification and actual API calls.

Create Meta-Task:

Open a new GitHub issue titled "Reflections on Task 56" to describe the process you followed.
In the issue body, provide a breakdown of the algorithm used, the steps taken, and any adjustments made during the transformation.
Attach the example OpenAPI specification and the corresponding series of API calls as a comment in the GitHub issue.
This meta-task serves as both documentation and a reference for future tasks.

By following these instructions, you can successfully transform a GitHub issue into an OpenAPI specification, generate corresponding API calls, and document your process in a meta-task for future reference and collaboration.

Here's the translation of the provided algorithm and instructions into Python code. This code assumes that you have access to the GitHub issue data and that you're using appropriate libraries for making HTTP requests and working with JSON data.
import requests

# Step 1: Access GitHub Issue Data
issue_url = "https://api.github.com/repos/meta-introspector/meta-meme/issues/66"
response = requests.get(issue_url)
issue_data = response.json()

# Step 2: Extract Issue Details
issue_title = issue_data['title']
issue_body = issue_data['body']
comments = issue_data['comments']['nodes']

# Step 3: Map to OpenAPI Task Specification
# Mapping issue title and body to OpenAPI title and description

# Step 4: Define Paths and Operations
# Analyze issue content and identify paths, methods, operation IDs

# Step 5: Create Request and Response Examples
# Utilize examples from the issue for request and response bodies

# Step 6: Define Schemas for Components
# If needed, create schemas for models or structures mentioned in the issue

# Step 7: Annotate with Metadata
# Add annotations like tags, documentation, security schemes

# Step 8: Translate to API Calls
# Implement API calls based on OpenAPI specification

# Print issue details
print("Issue Title:", issue_title)
print("Issue Body:", issue_body)
print("Comments:", len(comments))

# Print each comment
for comment in comments:
    author = comment['author']['login']
    comment_body = comment['body']
    created_at = comment['createdAt']
    print(f"Comment by {author} at {created_at}:")
    print(comment_body)
    print("-" * 40)

# Step 9: Document Reflections (Not implemented in this code snippet)
# Document insights, challenges, and transformation process

# Step 10: Create Meta-Task (Not implemented in this code snippet)
# Open a new GitHub issue describing the transformation process

Please note that this code is a simplified representation of the algorithm and instructions provided in the GitHub issue. You'll need to fill in the details for Steps 3 to 8 based on the specifics of the issue content and the OpenAPI specification. Additionally, Steps 9 and 10 involve manual actions on GitHub and are not implemented in this code snippet.
Make sure to install the necessary libraries (requests for making HTTP requests) before running the code.


Add systemd.timer as a run mode along with cron

Set default for periodic refreshes as a 'cron'
Alternatime 'systemd.timer' value of mcollective::periodic_runmode can be defined in hiera or puppet code.
commit 5fb16013809e4e995e6ef78fb4d8e6a89021dd8c
Author: Sergey Ivanov<EMAIL_ADDRESS>Date:   Thu May 11 21:01:51 2023 -0400
add runmode systemd.timer along with cron

Sorry, it was just 'puppet-lint --fix'. Changed to preserve code style and force-pushed to the branch.

I think #327 replaces this PR.

I think #327 replaces this PR.

Indeed, closing this PR.  Thanks!


add Resources section #272

closes #272
I took a quick swing at at least putting links to papers, etc. onto a Resources page. The formatting is extremely minimal and looks like this right now:

I'm not sure how fancy we should get. At minimum, we should probably link the titles to the papers like they do at https://differentialprivacy.org/resources/ (they also link the author names, which would be more work):

We could also try to use full on citations and a Sphinx extension where we keep all these in Bibtex format. Probably overkill, I'd say.
Under the "Papers" section I left comments from @Shoeboxam in parentheses. We can take these out or otherwise rework them.

This is ready for more review. I just did the following:

Remove Rényi and Concentrated
Make titles into links
Move the notes in parens to the side

Here's how it looks:


INFO 112 2022-03-30T19:43:33.321 postgres: starting PostgreSQL 13.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
INFO 112 2022-03-30T19:43:33.324 postgres: listening on Unix socket "/run/edgedb/.s.PGSQL.5432"
INFO 112 2022-03-30T19:43:33.324 postgres: database system was shut down at 2022-03-30 19:43:19 UTC
INFO 112 2022-03-30T19:43:33.325 postgres: database system is ready to accept connections
INFO 1 2022-03-30T19:43:35.823 edb.server: Connection established to backend database: edgedb
INFO 1 2022-03-30T19:43:44.213 edb.server: started 5 compiler worker processes
INFO 1 2022-03-30T19:43:44.234 edb.server: Serving on {<IP_ADDRESS>, ::}:5656
INFO 1 2022-03-30T19:43:44.236 edb.server: Serving admin on /run/edgedb/.s.EDGEDB.admin.5656

$ netcat -z -v localhost 15656
localhost [<IP_ADDRESS>] 15656 open

MacOS 12.0.1
EdgeDB CLI 1.1.1+24a93b7

Are you on an ARM Mac?  EdgeDB docker images don't work there unfortunately due to edgedb/edgedb-docker#24

Yes, it's ARM. Oh ok, that's unfortunate.

If you open Terminal.app with Rosetta it should work:

What has a x86 terminal to do with the ARM docker engine?

Docker Desktop is a multi-architecture binary, so I figured it would either run in emulation or not depending on the environment that originated the docker commands. But this is working for me on my M1 Mac both with and without emulation so I'm not sure.

This really worked for you? As far as I understand it, conceptually the docker command in Docker on Mac communicates with the Docker engine running inside a (ARM) virtual machine. In this ARM virtual machine are living all containers. The virtual machine is always an ARM architecture on a M1 mac, so it doesn't matter /how/ you communicate with it, whether be it via a docker cli running in x86 translation or not. It should not make any difference.
Although you can run x86 container on an ARM Mac (via --platform linux/amd64), it is then using inside the ARM virtual machine a qemu emulator, which is not very stable, nor feature complete, or fast. I mean there is also xbuild that allows you to build x86 images on ARM, but again this is terribly slow as it also uses qemu. So I'm trying to understand how this works for you when in theory it shouldn't.
It seems to be able to build our own ARM image from https://github.com/edgedb/edgedb-docker it's required that the apt server in https://packages.edgedb.com/apt needs to host ARM builds as well, or better: edgedb is building inside the docker container as well instead of relying on an apt server.


How to pass environment variables to build?

For example, if my crate uses sqlx, I might want to only perform offline build which is normally done with
SQLX_OFFLINE=1 cargo build

Here's what worked for me:
        rustPkgs = pkgs.rustBuilder.makePackageSet'
          { rustChannel = "1.56.1"
          ; packageFun = import ./Cargo.nix
          ; packageOverrides = pkgs: pkgs.rustBuilder.overrides.all ++
               [ (pkgs.rustBuilder.rustLib.makeOverride
                   { name = "mycrate"
                   ; overrideAttrs = drv:
                       { SQLX_OFFLINE = 1
                       ; }
                   ; }
               ) ]
          ; };


fix: bump sc4s/splunk versions

:tada: This PR is included in version 1.8.7 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Tried to use $message->embed() in the email template for the notification-email channel

Laravel Version: v5.4.6
PHP Version: 5.6.15
Database Driver & Version: mysqlnd 5.0.11-dev - 20120503

Description:
Can't send a notification with an image in the email template:
Undefined variable: message (View: ...\resources\views\vendor\notifications\email.blade.php)
You say in the docs that "To embed an inline image, use the embed method on the $message variable within your email template. Laravel automatically makes the $message variable available to all of your email templates, so you don't need to worry about passing it in manuall" but it seems not to work. If this isn't a bug, how can I add an embeded image in the mail template?
Steps To Reproduce:
Send an email via the email notification channel and try to use in the mail template the $message->embed() method.

+1 same issue after upgrade to 5.4

+1, also after upgrading to 5.4 (v5.4.6, PHP 7.1.1, MySQL 5.7.16)

+1, the same issue after updgade to 5.4.9

+1, the same issue after updgade to 5.4.9

+1, the same issue in 5.4.10

Started tracing the reason this does not work and the problem is even more deeply nested than just the notification emails. It is actually an issue with the way the markdown parser works.
Lets start by summing up my understanding of the notification-messages

The MailChannel's send method is called
It does some checking and then calls the buildView method to render the (markdown) views
The MailChannel's buildView method calls the render method on the markdown parser (found in Illuminate\Mail\Markdown)
The markdown parser renders both the html and text message into an HtmlString
The resulting data is then passed onto the mailer
The mailer sees the data is an HtmlString and appends it to the message
The mailer completes the message (calling the callback defined in the send method, etc)

The same actually goes for Mailables that use markdown, when the send method is called they also run a variation of step 4 - 7.
Since the markdown parser never sees the message in both cases, as soon as markdown is used for emails (Mailables or Notifications), the will never be able to use $message->embed or any related functionality.
The way I expect the markdown option to work is that (similar to regular blade-templates) the message is made available to the markdown parser through the data attribute instead of the message being pre-rendered to an HtmlString. This would mean a rewrite of the way the markdown functionality works to be handled inside the addContent section of the Illuminate\Mail\Mailer instead of rendering it inside the Mailable/Notification and sending the HtmlString. Maybe the easiest way is to "enhance" the views array with an additional "markdown" attribute (next to the html/text/raw attributes). As an added bonus this will make markdown message also available when not using Mailables/Notifications...

Just bumped into this as well. Going to revert to normal templates for now.

FYI, you can sidestep the issue until it's fixed by passing an email layout as a view at the end.
public function toMail($notifiable)
{
    return (new MailMessage)
        ->greeting('Hello!')
        ->view('layouts.email');

@aaronlord I still get Undefined variable: message after adding view
:(

Having the same issue still in  5.4.13, I think its too do with the Markdown parser.

Have this problem as well.
The $message here is a Illuminate\Mail\Message , which is rendered by https://github.com/laravel/framework/blob/5.4/src/Illuminate/Mail/Mailer.php
However, the notification emails such as password reset emails are Illuminate\Notifications\Messages\MailMessage, which are rendered by https://github.com/laravel/framework/blob/5.4/src/Illuminate/Notifications/Channels/MailChannel.php
I tried to add $message->viewData['message'] = $message; to the send() method of MailChannel but \Notifications\Messages\MailMessage do not have embed.
I think it is not a good idea to point to the same set of views for two incompatible classes.

I wrote a package that at the time of sending, it does the automatic embed
See if that helps.
laravel-mail-auto-embed

+1 same issue in 5.4.15

@eduardokum thank you, your package save my life!

@eduardokum Thank you

Having the same issue still in 5.4.21

Same issue in 5.4.22

Guys a $message value doesn't exist in the notification email template, that is by design.

@themsaid Can you elaborate how to make it available in custom markdown mailables?

@eduardokum you rock buddy. Thank you man :)

@eduardokum When I do ({{ asset("storage/$news->image") }}) all I am getting is a link to that image.

@eduardokum problem solved :smile: Just didn't realize the importance of this thing ![product] before ({{ asset("storage/$news->image") }}). Perfect :+1:

I wonder why I don't get images when queuing the mail.

@roshimon Please edit your message instead, getting lots of emails from this.
On topic: really wish there was less of a difference between mailables/views and markdown emails. You'd expect so anyway. Had to copy and tweak ALL Laravel markdown emails and layouts to my core app files to get the same, simple emails, but with support for the $message variable.

For anyone who following the doc to implement the following code in their email blade template

    Content: {{ $message->content }}

check the variable name 'message' to anything else (e.g clientMessage)

This works fine with an image named "logo.png" in the dir "public/images/":
{{ $message->embed(public_path().'/images/logo.png') }}


recipes post

Here is the blog post for the last coding club session. hope you like it :)

Thanks @ceciliamescobedo!! Well done and written :star: :star_struck:


cargo-make: 0.35.15 -> 0.35.16

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/sagiegurari/cargo-make/releases.
meta.description for cargo-make is: "A Rust task runner and build tool"
meta.homepage for cargo-make is: "https://github.com/sagiegurari/cargo-make"
meta.changelog for cargo-make is: ""
Updates performed

Rust version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.35.16 with grep in /nix/store/8y8jclwahc4pajqqfr1wpzjrnrhrjdda-cargo-make-0.35.16
found 0.35.16 in filename of file in /nix/store/8y8jclwahc4pajqqfr1wpzjrnrhrjdda-cargo-make-0.35.16

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
cargo-make

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/8y8jclwahc4pajqqfr1wpzjrnrhrjdda-cargo-make-0.35.16 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A cargo-make https://github.com/r-ryantm/nixpkgs/archive/e75ba1ab1766bbcdd6976f0eaf00710c79d16994.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/8y8jclwahc4pajqqfr1wpzjrnrhrjdda-cargo-make-0.35.16
ls -la /nix/store/8y8jclwahc4pajqqfr1wpzjrnrhrjdda-cargo-make-0.35.16/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

cargo-make

Maintainer pings
cc @xrelkd for testing.

Result of nixpkgs-review pr 185802 run on x86_64-darwin 1

1 package built:

cargo-make


Subscriptions list changing

I am subscribed to 120 channels, and it looks like i have the activityType === "All" on 100.
But it keeps removing and readding them to the database, the first 82 is in the original position but after that it skips 262 and that gap seems to be getting larger every time it runs.
The console log also has them being added as new most times, i haven't subscribed or unsubscribed to any channel since i ran it the first time.

So, to confirm, select count(id) from subscriptions; gives 120, but select max(id) from subscriptions; is 262? Similar situation for me actually. Looking into it.

Do you have any logs you could share?

So, to confirm, select count(id) from subscriptions; gives 120

No, the last 20 is skipped:
https://github.com/MarcelRobitaille/bbyen/blob/8d46e87675d7b6209243ac1a19640b8ddb4504f4/src/subscriptions.js#L24
The database have 100 subscriptions in it but after id 82 the next have id 374, and then it randomly skip numbers, the last have id 398

Do you have any logs you could share?

Not that i know off, but:

Ok, same thing over here. I'm on it.

Looks like it's fixed in acc1c100ff6a78d9a6ace7efee5e7cb4b90b3c5e.


Homepage paths component - fixed buttons colors

resolving #101
I changed the buttons setting to match Efi's directions.
So now the buttons work like this:

Mobile view - displayed in color
Desktop view - displayed in gray/white, and colored only when hovered.

Screenshot where I hovered over the middle button in desktop view:

@Drorka left one comment, let me know when you finish.
You did a great job, thanks 👏

@adirkandel thanks, I modified the code

@adirkandel pinging you to check the fixes

@Drorka approved but with one comment to complete 🙏

@Drorka approved but with one comment to complete 🙏

Thanks for the review, I committed your suggestion

@adirkandel I think this can be merged?

WELL DONE!!!!
@Drorka 👏 👏 👏 👏 👏


If applied this commit fixes the incorrect position of sign-up & login error messages on the home page

The Problem :

When you enter wrong credentials while logging in , the error messages get displayed in the SIGN-UP box rather than the login box as shown in the screenshot below.

This was happening because the sign-up form partial had the error handling code block at it's top but the login form didn't. And if we added the error handling block to that as well then as expected same errors were displayed on both the places.

Steps to recreate :

Try to login in the live copy with wrong credentials

The Solution :

As the Laravel documentation itself suggests that when you have 2 or more forms on the same page , you should name the error bags for better & correct error handling. Docs : section 3 of 'Manually creating validators' with the title "Named Error Bags" : https://laravel.com/docs/5.3/validation#manually-creating-validators

This commit overrides the sendFailedLoginResponse() method of the AuthenticateUsers trait in the LoginController to give the error bag a name.

Thanks @introwit!

For posterity, turns out there's a faster way to do this:
https://github.com/tightenco/symposium/commit/f7422d8d2cebf1d35313ee1d8235297aaae0f790

@mattstauffer seems like I missed that. Thanks , learned a new thing :) but just out of curiosity may I know is there any specific reason behind that reordering of the imported namespaces in that commit? Would help me learn. Sorry for the silly question though!

@introwit Sure! I prefer to sort the import block by alphabetical instead of by statement length. :)

Oh okay! Thank you Matt :) Will keep this in mind for the future contributions.


Update CI

Coverage remained the same at 93.617% when pulling 65b0e6b36d8b0b32c15a05d4ac1bced6639ac00b on update_ci into 627851aa6cdf812e0e0222b2d099deff24b83b00 on master.

Coverage remained the same at 93.617% when pulling 65b0e6b36d8b0b32c15a05d4ac1bced6639ac00b on update_ci into 627851aa6cdf812e0e0222b2d099deff24b83b00 on master.

Coverage remained the same at 93.617% when pulling 65b0e6b36d8b0b32c15a05d4ac1bced6639ac00b on update_ci into 627851aa6cdf812e0e0222b2d099deff24b83b00 on master.

Coverage remained the same at 93.617% when pulling 65b0e6b36d8b0b32c15a05d4ac1bced6639ac00b on update_ci into 627851aa6cdf812e0e0222b2d099deff24b83b00 on master.


add explicit doc to override default from parent

Without this, dartdoc walks up the inheritance tree and uses a doc from a parent class. In this case, the doc from the parent class doesn't apply to AnimatedComponent's createState().  (See http://docs.flutter.io/flutter/widgets/AnimatedComponent/createState.html)
@abarth is this an accurate statement for this method? We could be more bold here if we want. "Should not override..."

ptal @abarth

LGTM


Reimplement segmented display widget

Not measured it yet however I have serious concerns about the performance of the current segmented display implementation. Would be better to reimplement the widget using fonts and taking advantage of egui's text rendering stack.
The current implementation may also have remaining rendering issues in some cases.
Steps to be taken:

Create variable segmented display fonts (7, 9, 16-segment) with multiple variable axes: slant, weight, stretch, median, ...
Remove old geometry definitions
...

Neither egui nor FontForge supports variable fonts yet. Bummer.


[ODS-6531] Minor fixes and improvements

Changes:

Fixed a few broken links
Minor formatting improvements

Will add these changes in a different PR.


Added responsive navbar

Signed-off-by: Prakhar Agarwal<EMAIL_ADDRESS>Description
Adds responsive navbar for mobile view
Notes for Reviewers
Before:
https://user-images.githubusercontent.com/17739006/145674339-10de5c04-7fa8-4353-9cb8-420f12fda0ef.mp4
After:
https://user-images.githubusercontent.com/17739006/145674282-15c496c9-2f9f-46c5-8a03-3d10c7acb5b1.mp4
Signed commits

[x] Yes, I signed my commits.

@Prakhar-Agarwal-byte , i've recently made this change which would make sure we don't have this much style change being added to the commit. Can you rebase your PR and then update it so that we don't get these unwanted formatting changes ?

@sudo-NithishKarthik Done


java.lang.ArrayIndexOutOfBoundsException

I'm trying to set up and test a local instance of Conveyal on my MacBook. When launching the v5.9.0-3-g08bf467.jar compiled from the most recent GitHub version, I get an out-of-bounds error that I can't figure out.
I built the jar using the most recent version of maven with the command "maven package".  It appears to have compiled properly with the v5.9.0-3-g08bf467.jar showing up in the target/shared folder.  When I try to launch the .jar with the command: java -Xmx2g -jar target/shaded/v5.9.0-3-g08bf467.jar, I get the following error message:
[jamesdeweese@MacBook-Pro-2 analysis-backend % java -Xmx2g -jar target/shaded/v5.9.0-3-g08bf467.jar
/\\\\_/\\\\\\\_
/\///////\_/\///////////_
/\_/\_/\_____________
/\\\\\//\\\\\\
/\//////\_////////////\___
/\_//\_//\__
/\____//\_/\_______/\_
_/\_//\_//\\\\\\/
_///////////////////____
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at com.conveyal.r5.R5Main.main(R5Main.java:27)
Edit: (The hashmarks above spell out a cool-looking 3-D-esque "RS" in the terminal, but the tabs are lost in this post).
I was reluctant to post because I wasn't sure if this was the result of a mistake I made or an actual problem with the code.  But I've gone back over everything several times,  checking to make sure my mongodb instance was properly installed and running, that I used the right version of maven to build the jar and that the Java version is correct.  (I did have to upgrade from Java 8, which I was using for OTP, to Java 11, to correct a problem with classes from the newer compiler not working on the older version of Java.)

Hi @jamesdeweese,
It looks like you were running R5 (without required arguments). To run analysis (instead of R5) with the latest release, the command would be java -cp target/shaded/v5.9.0.jar com.conveyal.analysis.BackendMain -- the Readme should be updated with this command.
I haven't set this up locally from scratch in a while. You may find some of the troubleshooting by others at #245 helpful.


Machine win-7 went dead from monitoring website.

Machine win-7 went dead from monitoring website.

Machine win-7 is back online and ready for service, it was network adapter was hung and required to reboot.


TLS ECDH 2c: ECHDE in TLS 1.3 (client-side)

This task is to implement the ECDHE computation in the TLS 1.3 ephemeral key exchange (as it is in MVP, so client-side only) using PSA, regardless of the value of MBEDTLS_USE_PSA_CRYPTO (as that option is only for 1.2, and 1.3 should always use PSA).
See #2338 and #5291 for client-side support of ECDHE-ECDSA and ECDHE-RSA in TLS 1.2, and the description of #5317 for a comparison between the legacy mbedtls_ecdh_ API and PSA Crypto.
Note: On-the-wire format for ECDHE parameters slightly differs between 1.2 and 1.3, so the TLS-specific writing/parsing code will probably differ. It's still possible to look at the 1.2 code for inspiration, keeping that in mind.

[ ] Make sure the ecdh_psa_xxx fields in struct mbedtls_ssl_handshake_params are present when TLS 1.3 is enabled even if MBEDTLS_USE_PSA_CRYPTO isn't.
[ ] #5509
[ ] Change ssl_tls13_read_public_ecdhe_share() to use TLS-specific parsing code.
[ ] Change mbedtls_ssl_tls13_key_schedule_stage_handshake() to use psa_raw_key_agreement().

Depends on: #5291 (cleaned 1.2 code for re-use or inspiration).

I read about TLS Handshake 1.2 vs 1.3. In 1.3 version the number of transfers was reduced in order to speed up the process. As a result ClientKeyExchange message has been removed and in 1.3 public key is send together with Client Hello message in key share extension. The goal here is to implement ECDHE computation for  TLS 1.3.
So looking at the second bullet. ssl_tls13_generate_and_write_ecdh_key_exchange() has to populate key_exchange  field of a KeyShareEntry in a KeyShare structure. key_exchange has the following format:
      struct {
          uint8 legacy_form = 4;
          opaque X[coordinate_length];
          opaque Y[coordinate_length];
      } UncompressedPointRepresentation;

and it seem that it is consistent with the result of psa_export_public_key():
https://github.com/ARMmbed/mbedtls/blob/6ca6faa67e8ef792f05eedb12af5255a6641cd0a/include/psa/crypto.h#L779-L785
Does it mean that the output from psa_export_public_key() can be directly copied into the given buffer (for key_exchange) ?
When I was looking on current implementation for 1.3 (non-psa) this covers only one case for MBEDTLS_ECP_TYPE_SHORT_WEIERSTRASS group and MBEDTLS_ECP_PF_UNCOMPRESSED format:
https://github.com/ARMmbed/mbedtls/blob/6ca6faa67e8ef792f05eedb12af5255a6641cd0a/library/ecp.c#L721-L791

Does it mean that the output from psa_export_public_key() can be directly copied into the given buffer (for key_exchange) ?

Yes, that should be the case, both for Short Weierstrass curves (secp256r1, secp384r1, and secp521r1) and for x25519, so you shouldn't have any transcoding do to. (TLS 1.2 originally had two points formats: compressed and uncompressed, now compressed has been deprecated, and we never had full support for it anyway (we could write but not read it) - TLS 1.3 only has uncompressed.)
(Also, when you read "peers MUST validate each other's public value Q by ensuring that the point is a valid point on the elliptic curve.", that's not something you need to worry about: it happens automatically when using the value to compute the shared secret, no need for a separate check.)
I'm not yet fully familiar with TLS 1.3, so take what follows with a grain of salt, but the formats should be fairly similar except for the following differences:

As you pointed out, the names and more importantly ordering of the messages is different: in 1.2 (with ECDHE-RSA and ECDHE-ECDSA) the server sends its share first in ServerKeyExchange and the client replies in ClientKeyExchange; in 1.3 the client goes first with one or more shares in the the key share extension of its ClientHello, and the server responds in the key share extension of its ServerHello.
As a consequence, in TLS 1.2 the server has to encode which curve it chose while the client doesn't; in TLS 1.3 I think both parties have to encode the chosen/offered curve(s)
In TLS 1.2 when encoding the curve, we first need to encode the "curve type" (named or explicit), in TLS 1.3 only named curves are supported so the curve_type byte is not needed (also, terminology issues: NamedGroup in 1.3 is the equivalent of NamedCurve in 1.2).

So, except for the "extras" (namedcurve/group, possibly list structure, etc), the formats of the actual key shares should be identical between 1.2 and 1.3, and identical to the format PSA uses for inputs and outputs.


--omp-nthreads no longer maxing out at 8

Version: 20.1.0rc1
Without setting --nthreads or --omp-nthreads, I'm getting both set to nprocs.
The old formula for --omp-nthreads was max(1, min(8, nthreads - 1)).
This probably changed along with the massive configuration object.

Additionally, --nthreads is not being respected. I'm poking around a little...

Additionally, --nthreads is not being respected. I'm poking around a little...

I can't reproduce this locally. No idea what's happening.

We are running fmriprep 20.2.1, and we started noticing cpu core spikes which leads to out-of-memory issue (probably too many processes running and using too much memory as aggregate)

We are trying to set the max core to 8, and for the most part, fmriprep seems to be honoring this number, except at the end as the graph shows. (red line is at 800%)
cat <<EOF > multi_proc.yml
plugin: LegacyMultiProc
plugin_args: {maxtasksperchild: 1, memory_gb: 14, n_procs: 8, raise_insufficient: false}
EOF

Should I not use multi_proc.yml anymore?

I will try setting --nthreads and --omp-nthreads and see if it limits the CPU cores.

Are you able to identify the processes that are filling up those high CPU spots?
And there should be a .toml file that's in your scratch directory (and it's also printed out almost first thing in the logs). Could you post that?

--nthreads and --omp-nthreads doesn't seem to cure it? I am still seeing some occasional bumps.. (could be just some artifact of how we are collecting information though..)

I've increased the thread counts to 16 with 30gb for vmem (it's 2022 after all!) so it doesn't get killed at the end when the memory usage spike happens.

The process count spikes at the end seems to be caused by a large number of antsApplyTransforms processes.. I see 64 of them running near the end. We aggregate processes running within certain time window (it should be 30 seconds). So it's possible that, if tons of different processes gets run within 30 seconds windows, that might register as many concurrent processes when in face they are running separately in rapid succession. How long does each antsApplyTransform command run usually?
Here is one of those 64 commands that it ran.
antsApplyTransforms --default-value 0 --float 1 --input /scratch/hayashis/job_9779068/work/fmriprep_wf/single_subject_9002_wf/func_preproc_ses_1_task_rest_wf/bold_split/vol0185.nii.gz --interpolation LanczosWindowedSinc --output /scratch/hayashis/job_9779068/work/fmriprep_wf/single_subject_9002_wf/func_preproc_ses_1_task_rest_wf/bold_t1_trans_wf/bold_to_t1w_transform/vol0185_xform-00185.nii.gz --reference-image /scratch/hayashis/job_9779068/work/fmriprep_wf/single_subject_9002_wf/func_preproc_ses_1_task_rest_wf/bold_t1_trans_wf/gen_ref/sub-9002_ses-1_T1w_ras_valid_corrected_xform_masked_reference.nii.gz --transform /scratch/hayashis/job_9779068/work/fmriprep_wf/single_subject_9002_wf/func_preproc_ses_1_task_rest_wf/bold_reg_wf/bbreg_wf/concat_xfm/out_fwd.tfm --transform /scratch/hayashis/job_9779068/work/fmriprep_wf/single_subject_9002_wf/func_preproc_ses_1_task_rest_wf/sdc_estimate_wf/fmap2field_wf/vsm2dfm/sub-9002_ses-1_phase1_rads_phdiff_unwrapped_recentered_filt_demean_maths_fmap_trans_rad_vsm_unmasked_desc-field_sdcwarp.nii.gz --transform /scratch/hayashis/job_9779068/work/fmriprep_wf/single_subject_9002_wf/func_preproc_ses_1_task_rest_wf/bold_t1_trans_wf/bold_to_t1w_transform/tmp-5zu0a5pc/mat2itk_pos-002_xfm-00185.txt
The raw data used to draw the graph is found here > https://dev1.soichi.us/tmp/_smon.out It's just a list of json object at various time

How long does each antsApplyTransform command run usually?

It's resampling a single volume, so shouldn't take very long. I would expect <30 seconds. And these jobs are handled by an internal multiprocessing pool so will respawn as quickly as Python can manage.

The scratch directory gets wiped when the job finishes.. so I've updated the app to preserve it. I am rerunning the test job now. For me > https://brainlife.io/project/5d786647281b9525d8821011/process/6210043aff10f482da8ffaf0


itag for saving data?

pyspider version: 3.8.0
Operating system: Ubuntu
Start up command: pyspider --config config.json

Expected behavior
I am scraping users' profiles, but there is some data I only want to save the first time, and other information I want to save every time.
The way I would do this if they lived on different pages is I would pass in different itag values to self.crawl, and only crawl the summary page with an itag of their username, but crawl their updates with an itag of their username + last updated time.
But if they live on the same page, is there a way to add an itag to the return statement at the end of a block, so some data is only saved sometimes?
Actual behavior
My current workaround is that I am creating a callback for saving the summary, and a callback for saving the more frequent updates, then passing the current page's data into the save parameter of self.crawl, and crawling example.com. The different callbacks just save the data right away from the save.
However, I'd like a way to do this without making extra HTTP calls. Any suggestions?

If they lives on the same page, why wouldn't you update them(even didn't change) as long as the information had already crawled?

To save space. I have been using pyspider to get cleaner, more targeted crawls than other tools. I like not having to deduplicate it after the crawl. But there is nothing for that?

I don't understand how it would be duplicated. pyspider will update the data of certain URL, not insert a new one.

Oh I didn't realize that each URL was a unique key for data. Is there a way to prevent old data from getting overwritten? For example, I want to capture data before it gets deleted. I could do something hacky like use #=<timestamp> in the URL but maybe there is a better way

yes


[WIP] Remove the catch-up function from the test code

I checked late after seeing intermittent errors in some test codes when github's CI was executed due to the catch-up function.
I was able to remove the catch-up function from the test code to solve this problem.

If it fails in the CI, it's probably because there's a bug. Avoiding the call will not solve it, just hide it.

This is a problem caused by Mocha waiting only for the first Promise in the process of processing Promise chain.
So I plan to make an issue of this.

describe ('Test of Promise in Mocha', () =>
{
    before("begin", () => {
        function p1():Promise<void>
        {
            return new Promise<void>((resolve, reject) => {
                setTimeout(() => {
                    console.log('Promise 1');
                    resolve();
                }, 100);
            })
        }

        function p2():Promise<void>
        {
            return new Promise<void>((resolve, reject) => {
                setTimeout(() => {
                    console.log('Promise 2');
                    resolve();
                }, 200);
            });
        }

        function p3(): Promise<void>
        {
            return new Promise<void>((resolve, reject) => {
                setTimeout(() => {
                    console.log('Promise 3');
                    resolve();
                }, 300);
            });
        }

        return p1()
            .then(() => { return p2; })
            .then(() => { return p3; });
    });

    it ('it 2', () =>
    {
        console.log("it 2");
    })
});

describe ('Test of Promise in Mocha2', () =>
{
    it ('it 1', () =>
    {
        function p1():Promise<void>
        {
            return new Promise<void>((resolve, reject) => {
                setTimeout(() => {
                    console.log('Promise 1');
                    resolve();
                }, 100);
            })
        }
        function p2():Promise<void>
        {
            return new Promise<void>((resolve, reject) => {
                setTimeout(() => {
                    console.log('Promise 2');
                    resolve();
                }, 200);
            });
        }

        function p3(): Promise<void>
        {
            return new Promise<void>((resolve, reject) => {
                setTimeout(() => {
                    console.log('Promise 3');
                    resolve();
                }, 300);
            });
        }

        return p1()
            .then(() => { return p2; })
            .then(() => { return p3; });
    });

    it ('it 2', () =>
    {
        console.log("it 2");
    })
});

Since this is new, I made it proceed in issue #190


Update package for modern Sklearn (>=1.1) and other package requirements

Package requirements for the current implementation of wildwood do not allow for use with sklearn >= 1.1;
The private function  _joblib_parallel_args was removed, see link.
Looks like  _joblib_parallel_args, delayed in forest.py are affected. The error that I get is below:
  File "<stdin>", line 1, in <module>
  File "/Users/.pyenv/versions/3.10.11/lib/python3.10/site-packages/wildwood/__init__.py", line 2, in <module>
    from .forest import ForestClassifier, ForestRegressor
  File "/Users/.pyenv/versions/3.10.11/lib/python3.10/site-packages/wildwood/forest.py", line 22, in <module>
    from sklearn.utils.fixes import _joblib_parallel_args, delayed
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (/Users/.pyenv/versions/3.10.11/lib/python3.10/site-packages/sklearn/utils/fixes.py)

@imerad
I'm not that I follow what you're saying. The latest version on pypi is wildwood == 0.2; pip says it's up to date when I try to update. What am I missing?

@vtsanders, wildwood-0.2 on pypi is from two years ago, the joblib repair  @imerad mentioned is in the last two months.
I have tried the  pip install git+https://github.com/pyensemble/wildwood.git option
This produced the following error in a mamba based environment

error: Multiple top-level packages discovered in a flat-layout: ['wildwood', 'experiments'].

With an Ubuntu Python-3.10 environment the install command did not return error but the package was installed with the name UNKNOWN; the package is not importable.
For now I try working with a local copy of the project opened with PyCharm, in a new file in its tests folder.
I wanted to try wildwood after reading that it has native support for categorical features. I had good results with my initial tests and very happy   --mahmut

Sorry for the confusion. The update I mentioned only affected the git repository. As for the pip package, we will try to fix it shortly.

Highly related, I suppose?:
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes'

@markusloecher

Highly related, I suppose?: ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes'

Yes it is.

I would still love some advice on how to install wildwood, none of the suggestions worked for me.
Installing it directly from git is not successful:
error: Multiple top-level packages discovered in a flat-layout: ['wildwood', 'experiments'
Thanks!

Dear all, I just updated a new version 0.3 that should solve all these issues, hopefully

Thanks Stephane, I have tried the simple pip install with three different Python environments, all worked fine.
--mahmut


Ghost Followers example no longer works

Describe the bug
Ghost Followers example no longer works
To Reproduce
import instaloader

L = instaloader.Instaloader()

USER = "sergeysypalo"
PROFILE = USER

# Load session previously saved with `instaloader -l USERNAME`:
L.load_session_from_file(USER)

profile = instaloader.Profile.from_username(L.context, PROFILE)

likes = set()
print("Fetching likes of all posts of profile {}.".format(profile.username))
for post in profile.get_posts():
    print(post)
    likes = likes | set(post.get_likes())

print("Fetching followers of profile {}.".format(profile.username))
followers = set(profile.get_followers())

ghosts = followers - likes

print("Storing ghosts into file.")
with open("inactive-users.txt", 'w') as f:
    for ghost in ghosts:
        print(ghost.username, file=f)

Expected behavior
It should print a list of the ghost followers
Error messages and tracebacks
Fetching likes of all posts of profile sergeysypalo.
<Post ClO0OFNDz5D>
JSON Query to graphql/query: HTTP error code 401. [retrying; skip with ^C]
JSON Query to graphql/query: HTTP error code 401. [retrying; skip with ^C]
Traceback (most recent call last):
  File "C:\Users\Sergey\AppData\Local\Programs\Python\Python310\lib\site-packages\instaloader\instaloadercontext.py", line 352, in get_json
    raise ConnectionException("HTTP error code {}.".format(resp.status_code))
instaloader.exceptions.ConnectionException: HTTP error code 401.

Instaloader version
4.9.6

I just tried to get ghost followers using your script and it worked for me.
I believe this might be because Instagram is blocking your IP because Instagram suspects you of suspicious activity.

Guess yes, I have 2FA enabled, but previously it's just asked to provide a code for Google Authenticator, but lately instaloader failing on almost each and every task.


Roll engine 5dee035ee2d9..694300d32539 (14 commits)

https://github.com/flutter/engine/compare/5dee035ee2d9...694300d32539
git log 5dee035ee2d9a47123b72c0ac8d14b60e5c9fee1..694300d32539f22969d8a0e0ec9c1efbfa599125 --no-merges --oneline
694300d32 Provide a default pressure range if a MotionEvent does not have a device (flutter/engine#7737)
5dfa02949 Roll src/third_party/skia d336ebdea812..77a1a7e6182e (5 commits) (flutter/engine#7784)
dbb90a534 Roll src/third_party/skia 136cd7018a99..d336ebdea812 (4 commits) (flutter/engine#7781)
6933fcb2b Roll src/third_party/dart 174d6fec3d..fdfe40ea95 (27 commits)
f4a22d896 Roll src/third_party/skia ef26fcb13fc6..136cd7018a99 (1 commits) (flutter/engine#7778)
9c2f8726d Roll src/third_party/skia 5fc138c04690..ef26fcb13fc6 (1 commits) (flutter/engine#7775)
94e0e2c20 Roll src/third_party/skia fbe2e937c798..5fc138c04690 (1 commits) (flutter/engine#7774)
c3523d910 Roll src/third_party/skia 475134f2de57..fbe2e937c798 (1 commits) (flutter/engine#7773)
967460f83 Roll src/third_party/skia 1e1bc8de5671..475134f2de57 (1 commits) (flutter/engine#7772)
81cdd5491 Roll src/third_party/skia dafd65217b4e..1e1bc8de5671 (1 commits) (flutter/engine#7771)
06cc2e459 Roll src/third_party/skia cd9ef731fe75..dafd65217b4e (1 commits) (flutter/engine#7770)
ba0f38b1d Allow specifying the out directory prefix (flutter/engine#7762)
b2a93978b Roll src/third_party/skia 0b36acdad970..cd9ef731fe75 (1 commits) (flutter/engine#7769)
ddd224072 Roll src/third_party/skia 1f43a4359d7d..0b36acdad970 (1 commits) (flutter/engine#7768)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff (chinmaygarde@google.com), and stop
the roller if necessary.

Auto-roller completed checks. About to merge.


DUP - Archive old passes

DUP is currently storing 100s of thousands of expired passes - this ticket is to "archive" these passes so that it frees up some space in the system as well as improving security and privacy requirements. This ticket relates to #356 and may be part of that solution.
Acceptance Criteria:
Given
When
Then
Development Checklist:
[ ]
[ ]
[ ]
Workflows:
Dependencies

Relates to #356

Definition of Ready (Note: If any of these points are not applicable, mark N/A)

[ ]  User story is included
[ ]  User role and type are identified
[ ]  Acceptance criteria are included
[ ]  Wireframes are included
[ ]  Design / Solution is accepted by Product Owner
[ ]  Dependencies are identified (technical, business, regulatory/policy)
[ ]  Story has been estimated (under 13 pts)

Definition of Done

In progress:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] UI meets accessibility requirements
[ ] Unit tests are written
[ ] Work is traceable in GitHub
[ ] PR linked to ticket number
[ ] If needed/required - Dev adds flag/label to highlight any migration steps necessary prior to PROD deployment

Code review:

[ ] Code is peer reviewed and has passed CI/CD tests

QA:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] Code is potentially shippable to the production environment
[ ] Functional features have been tested and passed by QA
[ ] UI components tested by designer
[ ] Code is deployed to PROD when moved to 'done' column (unless requested otherwise by PO)

PO Review:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] Reviewed and approved by Product Owner

Notes:

Cam: Need to anonymize them and archive them by year, adding to a new table.

Cam: Not actionable until we determine when we want this to run and how far back (eg: if it runs on April 2024, what's the latest year that gets archived and removed from the active database?)

Idea:
Ticket 1: add a spike to clarify some things related to this ticket?
Ticket 2: first cut of archiving everything from 2021-2022
Ticket 3: Future: (THIS TICKET) Automating things would come out of the spike.

November 21, 2024 - DUP prioritization session. This ticket/issue was identified as medium-high priority.
Additional notes

For devs this is a higher priority > lots of passes in DynamoDB, which are not useful to current system and slowing things down. Moving historic passes into their own table will improve speed and reduce carrying costs.
With current structure, admin portal is parsing through a lot of passes/data & information that isn't needed.
linked to ticket 425

https://app.mural.co/t/bcparks2575/m/bcparks2575/1729124366479/8adea59d9e76fb3eb1e33395f1c1857a8c07fb7b

The work we discussed that is needed is captured in #384


Remove Hosts and File-based host list in favor of DiscoverProvider

We have DiscoverProvider interface for passing in a list of hosts to bootstrap. We also have a legacy mechanism File and Hosts to pass in a JSON file and a static list of hosts.
To do: replace File and Hosts with the equivalent DiscoverProviders, and remove the legacy ones.

Merged in c8f2ad4b720615334544f7851120bf9aa3f31306.


In addition to the provided obsevations, can we use extra info in the challenge?

Our team is interested in participating in the RoboTHOR Challenge. At runtime, in addition to the observations provided, our method would also require the wall structure, which we can extract from the 3D scan. We are wondering if it is eligible to encode this wall structure prior in the Agent class and submit to the challenge?  Thanks in advance!

Yes, it is perfectly fine to encode wall structures. However, we will not provide that information, and you need to infer that automatically.

Yes, it is perfectly fine to encode wall structures. However, we will not provide that information at test time, and you need to infer that automatically.

Thanks for letting know! We can infer the wall automatically using the 3d scan of the environmen. Will the 3D scan or 3D reconstruction be provided for the testing scenes?

No, they are not provided at test time.


Convention __n for 0 and negative values

As one and othersays: Shouldn't it be plurar (other) when the value is 0 or less as well ?
Would read:
"I own 0 cats."
instead of
"I own 0 cat."

This is somehow not supported anymore.
See https://tonicdev.com/574d4a7d85f76f1300abf1a5/574d4a7d85f76f1300abf1a6
i18n.__n('There is one monkey in the %%s', 'There are %d monkeys in the %%s', 0, 'tree');
Returns "There is one monkey in the tree"


UX: /log is now log-map, moorage name in title, duration formatting, links on cards

Description

/log/id is now log-map, log-details not used anymore
StaysMoorage: moorage name in title
MoorageArrivalsDepartures: moorage name in title
Moorage/Stays: duration in days+hours
Cards: add links to moorage (to, from)

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] Improvement/refactoring (non-breaking change that doesn't add any feature but make things better)

awesome work, thanks


Contribute to generating Open DigitalTwins Building classes

I would like to share my changes to allow your generator tool to parse the official Open DigitalTwins Building model (https://github.com/Azure/opendigitaltwins-building).
Thanks for your open tool

Hey @LaurentLapierre thank you very much for your contribution.
Ideally I think we'd like to refrain from checking in generated model sets, especially ones as large as the REC models.
If you could modify this to output the models to the bin folder as many other tests do that would be great.
Additionally, if there's unique behavior you feel would warrant having tests for on the generated models, then maybe use a subset of the REC models if possible that would exercise this specific behavior.
Please let me know if you have any thoughts or suggestions.

Hello @ms-mikeb,
I understand the decision to separate the code source and the use. I have done for my projects.
Please let me know if you want to a new pull request without REC models.
There is at least one commit that may interest you - commit 57180061 -> "Fix bug of empty description or display for EnumPropEntity"


Fix inconsistent CI workflows.

https://github.com/KitwareMedical/ITKHalideFilters/actions/runs/10426235433/job/28878788922
build-test-cxx suceeded on ubuntu-22.04, but failed on macos-12 and the other platforms were canceled. python-build-workflow succeeded on macos and windows, but failed on linux so the other platforms were canceled.
Some other notes in the comments of #8.

The issue on linux is a linking error due to Halide and glibc interacting in manylinux.
https://github.com/KitwareMedical/ITKHalideFilters/actions/runs/10874257586/job/30201463932#step:5:451
Notes at the end like undefined reference to<EMAIL_ADDRESS>or undefined reference to<EMAIL_ADDRESS>manylinux-2014 provides glibc 2.17 Halide claims support for 2.12+; so this might be some environment issue causing Halide to attempt to link to the wrong glibc.
https://github.com/pypa/manylinux?tab=readme-ov-file#manylinux2014-centos-7-based-glibc-217
https://halide-lang.org/docs/md_doc_2_building_halide_with_c_make.html#cross-platform

The issue on MacOS is due to Halide GPU only targeting CUDA. Should update CMake to use cuda or metal depending on the target platform.

The issue on Windows is a link error: https://github.com/KitwareMedical/ITKHalideFilters/actions/runs/10885362184/job/30202791711#step:6:774

https://open.cdash.org/builds/9907078/configure
I believe this is the root cause of the issues:

CMake Error: CMake can not determine linker language for target: HalideFilters
CMake Error in src/CMakeLists.txt:
Exporting the target "HalideFilters" is not allowed since its linker language cannot be determined

Halide 19 is intended to be published on PyPI, and will provide CMake config files. It doesn't make sense at this point to handle our own workarounds to force the manylinux or other Python builds to work when this is already being done (and partially complete) in the Halide 19 PyPI efforts.
Once that work is complete, the best way to handle the Halide dependency will be to pip install halide, and link to it with CMake.
Halide can be listed as a build dependency in the scikit-build pyproject.


Feature request: make layer-xxx: works even no native css layer support

One of well-known issue of tailwind is that you can't easily override classes via className, except you define the component classes in css. There's npm package tailwind-override solves that issue in runtime with high overhead.
We already have layer-xxx: syntax to support generate native css layer output. This solves the override issue, but it requires browser to support native css layers.
We can advance one small step: Instead of generate native css layer output, we can generate those rules with other rules by take care of the ordering to achieve the same effect.

You can use uno-layer-xxx: for unocss defined layers, see https://github.com/unocss/unocss#layers


Last element as point of reference

I've added support to use the last item as reference.
Ex:
item(tag='td', order=1)
it takes the first element
item(tag='td', order=0)
it takes the last element
item(tag='td', order=-n)
it takes the last element -n
Some sites don't have constant elements due to a extra information.  Then, it is easier to keep track using the last element as point of reference.

Have you tested the changes in list_() -> list() ?
I'm not familiar with that code.

Well, the change was to avoid shadow of the native list function.  I use pycharm, I used to have an alert for that, but not anymore.
Any case, I think that the reverting doesn't affect the function.  But, I think that it is clearly as it was before.
I will revert that part and I will do some tests.

@mancuniancol  I have pushed the change into 0.0.17, you can checkout and test.

Perfect.  I will.  I keep you posted.

I've been doing some test in my machine and it is working correctly reverting the list method to list_. I will upload that change this weekend


tutorialeditor: hidden popup text

On a smaller screen the popup text of the general/quiz side of the UI gets concealed by the main part.

This editor is best to use on laptops / computers with big displays


Two Factor Enabled Account Logging in without code

Hi all,
So I recall a few weeks ago when I tested an account with two factor authentication enabled I got the expected response from the login attempt i.e. it did not log in.
Today however I tried that again and I managed to log in successfully without any two factor challenge even though it is enabled on the account. Is this expected behaviour now?

This is not official Instagram repository. We don't know.

I am aware that this is not an official repository, I was just wondering if anyone else is experiencing the same behaviour.


Seems getting info of 'group' slot doesn't works (in R)

require(xgboost)
data(agaricus.train, package='xgboost')
train <- agaricus.train
group = rep(seq(1:3), length.out =  length(train[[1]]))
dtrain <- xgb.DMatrix(data = train$data, label = train$label, group = group)
works
labels <- getinfo(dtrain, 'label')
head(labels)
[1] 1 0 0 1 0 0
doesn't works
groups <- getinfo(dtrain, 'group')
Error en .local(object, ...) : xgb.getinfo: unknown info name group
low level works
labels <- .Call("XGDMatrixGetInfo_R", dtrain, 'label', PACKAGE = "xgboost")
head(labels)
[1] 1 0 0 1 0 0
low level doesn't works
groups <- .Call("XGDMatrixGetInfo_R", dtrain, 'group', PACKAGE = "xgboost")
Error: unknown field group
explicit setinfo
setinfo(dtrain, 'group', group)
[1] TRUE
groups <- .Call("XGDMatrixGetInfo_R", dtrain, 'group', PACKAGE = "xgboost")
Error: unknown field group

@tqchen any change for getinfo "group" ?
I just tried with no success (by modifying the R source code) :-(
> getinfo(dtrain, 'group')
Error in getinfo.xgb.DMatrix(dtrain, "group") :
  [16:58:17] ../..//amalgamation/../src/c_api/c_api.cc:488: Unknown float field name group

Stack trace returned 10 entries:
[bt] (0) /home/benesty_local/R/x86_64-pc-linux-gnu-library/3.4/xgboost/libs/xgboost.so(_ZN4dmlc15LogMessageFatalD1Ev+0x36) [0x7fc23a9d9996]
[bt] (1) /home/benesty_local/R/x86_64-pc-linux-gnu-library/3.4/xgboost/libs/xgboost.so(XGDMatrixGetFloatInfo+0x10e) [0x7fc23a9e04fe]
[bt] (2) /home/benesty_local/R/x86_64-pc-linux-gnu-library/3.4/xgboost/libs/xgboost.so(XGDMatrixGetInfo_R+0x51) [0x7fc23a9d6671]
[bt] (3) /usr/lib/R/lib/libR.so(+0xd2d3c) [0x7fc263f52d3c]
[bt] (4) /usr/lib/R/lib/libR.so(Rf_eval+0x7c5) [0x7fc263f8fab5]
[bt] (5) /usr/lib/R/lib/libR.so(+0x1136a6) [0x7fc263f936a6]
[bt] (6) /usr/lib/R/lib/libR.so(Rf_eval+0x5a0) [0x7fc263f8f890]
[bt] (7) /usr/lib/R/lib/libR.so(+0x112538) [0x7fc263f92538]
[bt] (8) /usr/lib/R/lib/libR.so(Rf_eval+0x5a0) [0x7fc263f8f890]
[bt] (9) /usr/lib/R/lib/libR.so(Rf_eval+0x5a0) [0x7fc263f8f890]

linked to
https://github.com/dmlc/xgboost/issues/1431
https://github.com/dmlc/xgboost/issues/574 (closure solution)


Failed to parse http response

I know there had a some title issues before, but I think it's not similar question, so I open a new issues:
I got a array of ID type in query, but apollo can not parse this array, when array is empty, everything is ok.
and if I change the name(type) of this array item from ID to String, it works! So could you know why does it happens?
the doc about array in schema.json:
{
  "name": "child_ids",
  "description": null,
  "args": [],
  "type": {
    "kind": "LIST",
    "name": null,
    "ofType": {
      "kind": "SCALAR",
      "name": "ID",
      "ofType": null
    }
  },
  "isDeprecated": false,
  "deprecationReason": null
}

and relatived json data is:
{
	"__typename": "PhotoType",
	"id": "818287",
	"url": "http://img.kid17.com/teacher/debug/kind_73/7fbdc3c95501992d745d694922e15cb5/pic/818287.jpg",
	"rotation": 0,
	"child_ids": ["35853", "35858"]
}

and the exception:

That odd, are you using the latest version?
As by default it supposed to be mapped to String

No, I'm still using version 0.4.1, and I will try latest version again.

I upgrade apollo to version 0.4.2, and everything becomes OK, that's great!

I upgraded and everything becomes OK.


Add back aria-hidden true to tabs for consistency.

Also updates tests to understand correct attribute removal
Addresses the consistency and testing issues raised in https://github.com/zurb/foundation-sites/pull/9916

@kball LGTM ( specially functionality as it doesn't break anything here )
Will await @Owlbertz to look into it though aka testing!

Phew .... Sigh of relief :D


Update embedded login dialog following UX review

Summary
Fixes #359

Occurred changes and/or fixed issues

add text to clusters list refresh button
make the login dialog more rancher login like

Don't use a generic prompt or tabs
allow user to toggle between local and dex options

The Auth Provider option is available even if my user is not enable to Dex login, this is because the oidcEnabled flag is true:


A web app like this should have a very low entry barrier so a login (as suggested by #3 OAuth) makes sense for feedback - but it does not prevent spam.

I agree, but I think we should be more specific about what specific sorts of filter to implement. I'm going to close this and appreciate more specific ideas or pull requests to reduce spam.


Fix several typos throughout code base

This addresses several typos in the strings throughout the code base (code list mappings, log messages, docstrings, etc.).

Lgtm

Lgtm

Then feel free to go ahead with the merge :-).
I have some more changes regarding the README.md in the pipeline, but those are a general improvement of the documentation and not related to this one here. From my point of view this MR is as ready as it will get :-).

I'm not near a computer and GitHub mobile refuse to rebase and merge

I'm not near a computer and GitHub mobile refuse to rebase and merge

Ah, now I understand the problem. I've rebased this change set on top of your current main, so no further rebasing needed. Should be possible to merge this by simply pressing the button. Of course, only do so when you're ready / available.


Asserting CONTRIBUTING guidelines.

This CONTRIBUTING.md file will help our fellow contributors to show by which mens they can contribute to this project

LGTM, would recommend public builds as well to enforce the yarn lint and yarn test recommendation.


feat: #1377 - reextract ingredients

What
It should be possible to extract ingredients without taking a new photo
More details : https://github.com/openfoodfacts/smooth-app/issues/1377#issuecomment-1108399793
Screenshot
https://user-images.githubusercontent.com/87010739/165115284-58d45791-16b5-4907-899e-7cb1b2718a34.mp4
Fixes bug(s)

#1377

Part of

#107

Codecov Report

Merging #1644 (b9b1efa) into develop (2ea0da3) will decrease coverage by 0.19%.
The diff coverage is 1.04%.

@@            Coverage Diff             @@
##           develop   #1644      +/-   ##
==========================================
- Coverage     8.86%   8.66%   -0.20%
==========================================
  Files          161     164       +3
  Lines         6623    6829     +206
==========================================
+ Hits           587     592       +5
- Misses        6036    6237     +201

Impacted Files
Coverage Δ

...h_app/lib/cards/category_cards/abstract_cache.dart
0.00% <0.00%> (ø)

...p/lib/cards/category_cards/asset_cache_helper.dart
0.00% <0.00%> (ø)

...p/lib/cards/category_cards/raster_async_asset.dart
0.00% <0.00%> (ø)

...oth_app/lib/cards/category_cards/raster_cache.dart
0.00% <0.00%> (ø)

..._app/lib/cards/category_cards/svg_async_asset.dart
0.00% <0.00%> (ø)

...smooth_app/lib/cards/category_cards/svg_cache.dart
0.00% <0.00%> (ø)

...t_cards/knowledge_panels/knowledge_panel_card.dart
0.00% <0.00%> (ø)

...s/product_cards/smooth_product_card_not_found.dart
0.00% <0.00%> (ø)

.../smooth_app/lib/data_models/onboarding_loader.dart
0.00% <ø> (ø)

...mooth_app/lib/data_models/product_query_model.dart
1.38% <ø> (ø)

... and 30 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a3a02d1...b9b1efa. Read the comment docs.

It looks great 👍 @cli1005
I would add a little space between the field and the text, and possible curve the top of the card (if possible, not a big deal)

@g123k @M123-dev @monsieurtanuki does that look good to you ?

It looks great 👍 @cli1005 I would add a little space between the field and the text, and possible curve the top of the card (if possible, not a big deal)

✅


Fix componentDidCatch and getDerivedStateFromError error argument typing!

Issue
componentDidCatch and getDerivedStateFromError in Component2 used to be passed the error argument and it was not re-dartified so we would receive a JsObject in ddc or NativeJavaScriptObject in dart2js so all typing and properties were not available. see: https://github.com/dart-lang/sdk/issues/36363 for more detail.
Fix
@greglittlefield-wf and I were able to find a solution that is able to re-dartify the error passed in from react and then forward that dartified error to the dart method of Component2.
How it works:

Error/Exception is thrown from dart
React catches the error JS side
JS passes error as argument (this does not get dartified)
Dart then passed the error to JS to re-throw the error
Dart wraps that call to throw with a try-catch (this re-dartifies the error)
In the catch on the dart side we then pass the dart error to the component method

QA

[x] Run pub run build_runner serve example, go to http://localhost:8080/test/react_test.html click Trigger Error

[x] "The Code is a 2" is above the button
[x] There is a non-error logged stack trace in the console that matches the React logged stacktrace.

[x] Run pub run build_runner serve -r example, go to http://localhost:8080/test/react_test.html click Trigger Error

[x] "The Code is a 2" is above the button
[x] There is a non-error logged stack trace in the console that matches the React logged stacktrace.

[x] CI Passes

QA +1


Small UI/UX feature for Button

Is your feature request related to a problem? Please describe.
When It comes to the UI/UX of the website it looks buggy
Describe the solution you'd like
I would like to add a proper button, The website looks buggy right now
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context

Go on buddy, assigning you this task

closing this issue!


Precompute summation indices for the model expressions

Pull request details
Describe the changes made in this pull request
To avoid using conditionals inside of summations in the expressions, we precompute them.
We also compute the duration.
This speed up the code, but maybe is it readable enough?
List of related issues or pull requests
Related to #294
Collaboration confirmation
As a contributor I confirm

[x] I read and followed the instructions in README.dev.md
[x] The documentation is up to date with the changes introduced in this Pull Request (or NA)
[x] Tests are passing
[x] Lint is passing

PS, the profiling has to be run locally with the EU data.

In the SpineOpt model (MOPO project), they follow this same strategy, meaning that they have a function that creates the indices for the constraints as a dictionary and then use it to create the constraints in the model. That's the basic idea, but they have a different structure for the temporal resolution, so their functions to find the indices are quite different.
We can learn how to run the profiler during tomorrow's meeting and make a decision. The readability is not looking as bad as I expected 😄

The readability is not looking as bad as I expected 😄

So far...


eyegaze

How would one implement a axis that shows the direction of the eyes are looking at ?

You need to track a stable reference point to measure the offset of the pupil from. Like the eye corners.
I haven't managed to do this, it's quite difficult, especially if you want high accuracy.

Hi Tristan, sorry for my late reply.
Could i not just draw a line instead of a circle for the eye? I'm doing this currently (in main.cpp)
line(debugFace,rightPupil,rightPupil+cv::Point(-15,-2),CV_RGB(255,5,255),2,8,0,0.5);
line(debugFace,leftPupil,leftPupil+cv::Point(-15,-2),CV_RGB(255,5,255),2,8,0);
The only problem i have is that it moves the wrong end of the line. It moves the pupil, not the other end . Ideally i would have the centre in the eye, and then the other end of the line would show the pupil movements.
Would appreciate your help!

Hey atv2016, I've looked into this as well so thought I'd share what I found.
eyeLike doesn't really track it at the moment, I think that's what's meant by "I haven't managed to do this".
Gaze tracking would be very useful though.  Last time I checked (several years ago) there were a few teams claiming working proprietary implementations but nothing freely available.  I think there are some vague papers written seemingly intentionally in broken Engrish available on Google Scholar.
I think you're gonna have to (re)invent the wheel if you want gaze tracking.    Please share it with us if you do :)

Yes, but currently i put a line instead of a circle. That line moves with the pupil. All i need is the other end of the line to move with the pupil? That shouldn't be to hard?
Or am i mistaking how to implement this? All i need is a line in which way the user is looking, or maybe a cone later on.

It is hard, that's the thing.  Not saying pupil detection is easy though.
Pupil detection and gaze tracking are two different problems.
Tristan has worked hard to provide this free software library which includes pupil detection but is yet to do gaze tracking (the disclaimer in the readme says this clearly).  Tristan did not invent this method, it's based on others' research.
There are papers on gaze detection, but I don't believe there's an open source implementation for average quality video yet.  Maybe you'll be the one to do it?  :)

https://scholar.google.com/scholar?as_ylo=2016&q=gaze+detection+opencv&hl=en&as_sdt=0,5
http://sci-hub.cc
Tell us about it once you've done it

If you've got the skills and the time to implement this and are willing to release it back to the community, I'm sure you'll find willing help.  The OpenCV community would be a good place to look for collaborators.

Have you considered integrating the work found here?   The project, aitrack, would be able to provide all of the reference points necessary to accurately determine gaze.

Have you considered integrating the work found here?   The project, aitrack, would be able to provide all of the reference points necessary to accurately determine gaze.

I'm not really working on this project anymore, but one issue is that the reference point and pupil tracking both need to be very accurate to get reasonable gaze point estimation. I'm not sure eyeLike is even accurate enough at pupil location to be that useful, and most 6dof head pose systems aren't accurate enough for this use case either. It would also take a bunch of work to integrate.

I'm not really working on this project anymore, but one issue is that the reference point and pupil tracking both need to be very accurate to get reasonable gaze point estimation. I'm not sure eyeLike is even accurate enough at pupil location to be that useful, and most 6dof head pose systems aren't accurate enough for this use case either. It would also take a bunch of work to integrate.


Coder Bin Path

Hey, thank you for taking your time to read this.
Due to security policies in place, we are having issues using this extension without getting exceptions passed. It appears the fetchBinary() method in storage.ts is calling getBinaryCachePath() which uses path.join(this.globalStorageUri.fsPath, "bin") that returns %APPDATA%\Code\User\globalStorage\coder.coder-remote\bin\. Is it possible to either be able to choose where this bin gets stored/checked, or possibly update the checkBinaryExists() to check for system/user environment variables to see if the Coder CLI is already in the PATH? This would alleviate a lot of headaches if we could either choose the path to store the bin folder/file or be able to check to see if we already have a folder specified in our environment variables for the Coder CLI. No issues if the extension needs to update the Coder CLI in the environment variable PATH.
Please let me know if you have any thoughts/potential workarounds, thank you very much!

I was just about to create this issue but saw it already exists!
In the JetBrains plugin we added a setting that lets you configure the path of the binary so we should do the same here.
One use case seems to be configuring that to be some restricted location like in Program Files so when the binary is out of date and the plugin is out of date but cannot write to the provided directory we added two options:

Disable the version check, always use the provided binary
Fall back to the default, writable location


Prepare for release v2021.03.11

ProductLine: KubeDB
Release: v2021.03.11
Release-tracker: https://github.com/kubedb/CHANGELOG/pull/34
Signed-off-by: 1gtm<EMAIL_ADDRESS>

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: 1gtm:x: kodiakhq[bot]You have signed the CLA already but the status is still pending? Let us recheck it.


Calculating add. free space

Calculating free space instead of adding a fixed amount of free space.

Thank you for the PR. The idea with adding free space to the end is only to allow the system to boot the first round. I ran into some issues with there being zero free space so the intent is to add a minor amount to let the system breath. 5% could be rather large on things like emulator setups so I prefer it to stay a static, small size. But I appreciate your efforts.


Error - testing Cluster setup on Windows

I am trying to test cluster setup on Windows, and cross platform with RaspberryPi.
I have made changes to the config file
[meta]
  bind-address = "<host name>:8088"
  http-bind-address = "<host name>:8091"
[http]
 bind-address = "<host Name>:8086"

Then I startup first node in Windows, and trying to join it from a node running in another PC. I keep getting [metastore] 2016/02/17 11:21:04 waiting for join peers to match config specified. found [<host1>:8088 <host2>:8088], config specified [<host1>:8091] error (added as part of #5602, in store.go)
I tried joining from a node running on raspberry pi, and got same result. My question is, is it specific to Windows? Any pointers where to start debugging?

@rossmcdonald I tried that, now I am getting different type of errors..
[meta] 2016/02/20 00:02:31 <IP_ADDRESS> - - [20/Feb/2016:00:02:31 +0530] POST /join HTTP/1.1 500 25 - Go 1.1 package http 22d98646-d737-11e5-804d-000000000000 2.0032ms
[meta] 2016/02/20 00:02:31 store not open

[meta] 2016/02/20 00:02:31 <IP_ADDRESS> - - [20/Feb/2016:00:02:31 +0530] POST /join HTTP/1.1 503 25 - Go 1.1 package http 22d91825-d737-11e5-8047-000000000000 1.9989ms
[meta] 2016/02/20 00:02:31 no leader

[metastore] 2016/02/20 00:25:11 [ERR] raft: Failed to make RequestVote RPC to Batman-G750:8088: dial tcp <IP_ADDRESS>:8088: ConnectEx tcp: i/o timeout
[metastore] 2016/02/20 00:25:12 [WARN] raft: Election timeout reached, restarting election
[metastore] 2016/02/20 00:25:12 [INFO] raft: Node at WIN-RSQ0FRMKUTB:8088 [Candidate] entering Candidate state
[metastore] 2016/02/20 00:25:12 [DEBUG] raft: Votes needed: 2
[metastore] 2016/02/20 00:25:12 [DEBUG] raft: Vote granted from WIN-RSQ0FRMKUTB:8088. Tally: 1
[metastore] 2016/02/20 00:25:12 [ERR] raft: Failed to make RequestVote RPC to Batman-G750:8088: dial tcp <IP_ADDRESS>:8088: ConnectEx tcp: i/o timeout
[meta] 2016/02/20 00:25:13 no leader

@mvadu What do your -join parameters look like for each node?

I have this influxd.exe -config influxdb.conf -join WIN-RSQ0FRMKUTB:8091,Batman-G750:8091 on both hosts. Using conf file with respective host names.

I think #5706 is not working on Windows. Setting top level hostname is not propagating to down levels. This leads to below set of errors on Windows, even without any cluster setup. Just influxd.exe -config influxdb.conf is giving me these errors:
2016/02/20 20:04:44 InfluxDB starting, version 0.9, branch unknown, commit unknown, built unknown
2016/02/20 20:04:44 Go version go1.4.3, GOMAXPROCS set to 8
2016/02/20 20:04:44 Using configuration at: influxdb.conf
[meta] 2016/02/20 20:04:44 Starting meta service
[meta] 2016/02/20 20:04:44 Listening on HTTP: [::]:8091
[metastore] 2016/02/20 20:04:44 Using data dir: ./influxdb/meta
[metastore] 2016/02/20 20:04:45 Node at BATMAN-G750:8088 [Follower]
[metastore] 2016/02/20 20:04:47 Node at BATMAN-G750:8088 [Leader]. peers=[BATMAN-G750:8088]
[metaclient] 2016/02/20 20:04:47 failure getting snapshot from :8091: Get http://:8091?index=0: dial tcp :8091: ConnectEx tcp: The requested address is not valid in its context.

Clustering has been removed from the open source version so I'm going to close this issue. If you need an open source solution for high availability, please try the relay. Thanks.