pre-processing images

Hi,
I'm trying to understand how things have changes from caffe to caffe2, and I've opened an issue there but I have a few separate questions for this repository,
and in specific the pretrained example, especially lines 55 to 87.

I see you calculate and normalise each image, regardless of the network (which used to be the case?)
You split the data in three separate channels (B, G, R?) but then you append it all in one vector?
are those tensors single dimensional because of the network, or am I misunderstanding it?
how can I upload the data to a GPU tensor since I'm doing all the opencv pre-processing using cuda?

Many thanks (and have a nice weekend!),
Alex

Perhaps it's helpful to summarize the steps taken in pretrained.cc:

Scale the image so that the shortest side of the image equals FLAGS_size_to_fit.
Crop to the image center so that both width and height equal FLAGS_size_to_fit.
Convert image color values from uchar (CV_8UC3) to float (CV_32FC3), assuming 3 color channels, and subtracting 128, so the mean color value is closer to 0.
Converting from NHWC to NCHW, which you could interpret as going from 1 image with 3 BGR colors per pixel to 3 BGR images with one color per pixel.

To answer your questions:

I see you calculate and normalise each image, regardless of the network (which used to be the case?)

That's correct. I'm making a lot of assumptions here, which might cause problems. However most ImageNet models work best on square images of size 224x224, with mean float values between -128 and 128, 3 color channels in NCHW order.

You split the data in three separate channels (B, G, R?) but then you append it all in one vector?
are those tensors single dimensional because of the network, or am I misunderstanding it?

So that's the NCHW order, which is default with Caffe2. Getting the order wrong usually turns the image into noise resulting in predictions like "window screen". Take a look at Caffe Prefers CHW Order for a more detailed explanation.

how can I upload the data to a GPU tensor since I'm doing all the opencv pre-processing using cuda?

In my experience it's not necessary to store input data in a TensorCUDA, assuming this is copied automatically by the operator. However you can copy a TensorCPU into a TensorCUDA blob like this:
TensorCPU tensor_cpu(...);
TensorCUDA tensor_cuda = workspace.CreateBlob("input")->GetMutable<TensorCUDA>();
tensor_cuda->ResizeLike(tensor_cpu);
tensor_cuda->ShareData(tensor_cpu);

@leonardvandriel Many thanks, I haven't had time yet to go through all that, but once I will I'd like to push a pre-processing example if that is OK?
The reason why I am asking about TensorCUDA is because I do all the pre-processing on the GPU, so I don't see the reason to upload the image on the device then copy it on the host only to copy it back to the device again.
I'll get back to you on this tomorrow.

Can you explain in more detail what you mean by "pre-processing on the GPU"? There's indeed no point in copying things to and from the host, but it don't think that's the case in any of the examples.

No your examples don't use the GPU, the issue I've posted in caffe2 uses OpenCV's cuda versions and the GpuMat matrix, which is being used more commonly with opencv >= 3.0

got it, thanks for explaining that. I've been using the older OpenCV 2.4, which was default on homebrew. Would be nice to have an example that uses GpuMat.

I'll try to do a pull request as soon as I've finished this.
I've asked on caffe2 how to set a Tensor<CUDAContext> because I can't find any examples or instructions, in either the source code or else where

Hi Leonard, I'm back with some code.
Sadly it seems to throw an assertion of wrong dimensions. I'm copy-pasting, assuming you may be able to help. Feel free to add to the repository, or I can do a pull request if you prefer.
caffe2::TensorCPU make_caffe_tensor::operator()(
                                                 const cv::cuda::GpuMat & image,
                                                 unsigned int channels,
                                                 cv::Size geometry
                                               )
{
    if (image.empty()) {
        throw std::runtime_error("empty image param");
    }
    // resize to input geometry
    cv::cuda::GpuMat resized;
    if (image.size() != geometry) {
        cv::cuda::resize(image, resized, geometry);
    }
    // crop image if needed
    cv::Rect crop((resized.cols - geometry.height) / 2,
                  (resized.rows - geometry.width) / 2,
                  geometry.height,
                  geometry.width);

    // convert colour type if needed
    cv::cuda::GpuMat sample;
    if (resized.channels() == 3 && channels == 1) {
        cv::cuda::cvtColor(resized, sample, cv::COLOR_BGR2GRAY);
    }
    else if (resized.channels() == 4 && channels == 1) {
        cv::cuda::cvtColor(resized, sample, cv::COLOR_BGRA2GRAY);
    }
    else if (resized.channels() == 4 && channels == 3) {
        cv::cuda::cvtColor(resized, sample, cv::COLOR_BGRA2BGR);
    }
    else if (resized.channels() == 1 && channels == 3) {
        cv::cuda::cvtColor(resized, sample, cv::COLOR_GRAY2BGR);
    }
    else {
        sample = resized;
    }
    // convert to float and normalize in one operation
    cv::cuda::GpuMat mfloat;
    if (channels == 3) {
        sample.convertTo(mfloat, CV_32FC3, 1.0, -128);
    }
    else {
        sample.convertTo(mfloat, CV_32FC1, 1.0, -128);
    }
    // split to channels
    std::vector<cv::cuda::GpuMat> input_channels;
    cv::cuda::split(mfloat, input_channels);
    // allocate => channels * float matrix size
    std::vector<float> data(input_channels.size() *
                            (mfloat.size().height * mfloat.size().width));
    // iterate, download and append
    for (auto & mat : input_channels) {
        cv::Mat tmp(mat.size(), mat.type());
        mat.download(tmp);
        data.insert(data.end(), (float *)tmp.datastart, (float *)tmp.dataend);
    }
    std::vector<caffe2::TIndex> dims({1, sample.channels(), sample.rows, sample.cols});
    caffe2::TensorCPU tensor(dims, data, NULL);
    return tensor;
}

My problem is that the above throws an assertion from the tensor constructor:
[enforce fail at tensor.h:128] values.size() == size_. 301056 vs 150528

AFAIK this is an issue of miss-matching dimensions?

Hi @leonardvandriel
I've managed to copy GPU pre-processed data from cv::cuda functions to a TensorCUDA but from what I can see, the Predictor only accepts TensorCPU.
Do you know how I can force the Predictor to run a CUDA tensor instead?

The Predictor is only a convenience class for setting up a workspace and running the model, see predictor.cc. I believe the use of Predictor is even discouraged.
Take a look at predict_example in mnist.cc, which implements a predictor from scratch.

@leonardvandriel Thanks a ton Leonard, you have been really helpful! This is where things seem to be getting even more complex; I see from that example that you use a TensorCPU which I can replace with a TensorCUDA following all the pre-processing on the GPU.
However, when I tried to add init_model.mutable_device_option()->set_device_type(CUDA); in my code, the actual application SEGFAULTs. My guess it is that option that sets the network to use the GPU instead of the CPU?
It is my understanding that not all models are built for CUDA and some will only run on CPU?
If that is the case, then surely there is a way to find if the model(s) support CUDA?
I'm starting to think that in addition to your tutorials, a few wrappers for caffe2 might be needed.

Hi @alexge233, SEGFAULTs usually indicate a bug or at least the absence of proper input checking. Have you tried to run the application with lldb (macOS) or gdb (Ubuntu) to see where things go wrong?  Perhaps it all runs fine on the last v0.8.1 tag.
And what do you exactly mean by "not all models are built for CUDA"? If an operator does not have a CUDA implementation, it should give a proper error message.

@leonardvandriel I didn't put it under gdb, but what surprises me is that removing the line init_model.mutable_device_option()->set_device_type(CUDA); stops it from happening.
So if I set the device type to CUDA, and use the code you posted, I should be able to forwrd propagate a CUDA tensor in the network using CUDA only?
I'll have a look next week and see what went wrong.
Also, I was under the impression that some models are trained for CPU and some for CUDA, is this wrong? Not that it is relevant to the question, but useful to know.

The exact error I got is an assertion triggered during my unit test (catch.pp):
FAILED:
  {Unknown expression after the reported line}
due to unexpected exception with message:
  [enforce fail at predictor.cc:27] blob->template IsType<TensorCPU>(). Blob is
  not a CPU Tensor: data

It seems like I've got to abandon the use of Predictor and move on to something else.

The idea is that for  every operator there are multiple implementations, one for each type of hardware (referred to as "device"). So by specifying set_device_type(CUDA), you make the complete model run on the GPU, while if you leave it out it will run on the CPU. Operators usually require input tensors to reside on the hardware they are on, so that's why you get the error not a CPU Tensor: data. There are Copy operators that copy to and from the GPU, but usually you won't need those. Indeed Predictor is not GPU-friendly and it a simple convenience class anyway; just implement that yourself (see mnist.cc predict_example()).
As for the crash: Adding set_device_type(CUDA) runs the complete model on a different hardware using completely different implementation of operators for that hardware. It's not uncommon that models work fine without it, on CPU, and then crash when you add it. It is often an indication that something is wrong in your code, so usually leads to an improvement overall. That being said, I've had some weird crashes with changes in Caffe2 in the past weeks, which they now seem to have solved, so make sure to update to the latest Caffe2 master.

@leonardvandriel Leonard thanks so much, I wouldn't have figured this out without your help.
I've managed to construct, load and execute CUDA versions of both a manually created tensor and the networks. Sadly the Copy operators caffe2 has are very unclear as to how they are to be used. For example, trying to copy from cv::cuda::GpuMat has a missing template specialisation. Trying to get help from the caffe2 developers has been very hard.
My last two questions are the following:

I see that some models name their output layer differently. I know I can get the blob names, but what is the safest way to get the output blob name? looking into the "predict" file?
I see a RunNet and RunNetOnce and the only documentation on it seems to be for Python (https://caffe2.ai/docs/tutorial-basics-of-caffe2.html). I intend to keep the network alive for the duration of my application (it will be used thousands of times). Should I be using RunNet instead? Do you have any examples of how I should be using it?

Again, tons of thanks; I will upload a full example if you want me to, showing how all of the pre-processing can be done using the GPU only.

You know what's driving me crazy?

I've looked at the mnist.cc example, and I even went through the predictor.cc code in detail
Every time I run the Predictor examples using the CPU I get what one would consider "correct" results
Every time I run the code which uses CUDA and not Predictor I get what appear to be random or "wrong" results.
I've decided to upload the code in a repository, and hopefully make it work as best as possible.
Right now I've committed only the pre-processing methods, but soon I'll add more.
The methodology I'm following is:

caffe2::NetDef run_net;
caffe2::NetDef init_net;
caffe2::Workspace workspace;

squeezenet::squeezenet(std::string init_file,
                                        std::string run_file)
: workspace("squeezenet")
{
CAFFE_ENFORCE(ReadProtoFromFile(init_file, &init_net));
CAFFE_ENFORCE(ReadProtoFromFile(pred_file, &run_net));
init_net.mutable_device_option()->set_device_type(caffe2::CUDA);
run_net.mutable_device_option()->set_device_type(caffe2::CUDA);
}

std::vector<result> squeezenet::predict(caffe2::TensorCUDA input)
{
    auto data = workspace.GetBlob("data")->GetMutable<caffe2::TensorCUDA>();
    assert(data);
    data->ResizeLike(input);
    data->ShareData(input);
    workspace.RunNet(run_net.name());
    auto out = workspace.GetBlob("softmaxout");
    assert(out);
    auto softmax = out->Get<caffe2::TensorCUDA>();
    return label(caffe2::TensorCPU(softmax));
}

I've managed to solve my problem and answer my questions.
Again many thanks, I'll post a repository I've built for this, feel free to copy-paste if you need!

@leonardvandriel I've uploaded the GPU stuff using OpenCV here: https://github.com/alexge233/espresso-maker
Feel free to copy-paste :-)

Awesome, thanks for sharing the code. I'll see if I can incorporate in the mnist or imagenet examples.


how can I know the client was disconnnet

When I use the OnDisconnect function ,I found it do not work when clients dissconneted

*SocketV4.OnDisconnect does works


No fan with RaspiOS 64 bit Bullseye

Since updating RaspiOS 64 bit from Buster to Bullseye, the fan no longer comes on. It worked fine in Buster. Is there a way I can update the fan script to work with Bullseye? Thanks!

I was able to get it manually going by changing the path in fan.py from
#! /usr/bin/python
to
#! /usr/bin/python3
I can hear the fan speed changing based on my CPU load, so it seems to be working properly for now with this change.

fan.py doesn't automatically start on login, so I've copied the script to /usr/local/bin/fan.py and configured i3 to run it at login, which seems to be working. The reboot/power off button work fine.


No search results found error

Am trying simple questions like "What is Google", still gives the same error.

I ran into a similar situation on Windows. But it works fine on Mac OS.
Here is my answer:
To fix it, you can use a custom Prompt format of the webchatgpt plugin.

Webchatgpt will require you to log in to MaxAI.me
and then they require you to install the MaxAI.me plugin.
(PS: You may need to pay and open the custom Prompt format. I am not sure. I paid for it but won't pay for it next time.)
Come back to webchatgpt; you can set your custom Prompt format, which you can generally use.

But it drains my request count of 600 requests per hour with no results. The whole project feels a bit like a con to me.


Create New Connection wizard - Generic driver won't allow Finish

This is a minor issue.
The generic drivers do not allow the user to click Finish if one or more "required" fields on the General page have not been entered.
On the last page of the wizard, it is not clear to the user what action is needed if Finish is still disabled. Either the driver should store an empty value for all required fields and allow the user to finish creating the connection (even if this connection cannot possibly ever connect), or the Next button should be disabled on the first page if a field there is required. The former behaviour seems to closest to the behaviour for custom drivers (at least the MySQL driver allows clicking through and hitting Finish).
As a very minor side note on the wizard behaviour, it seems to me that as soon as a driver has all information necessary to complete the connection, then the Finish button should be enabled immediately. For example, the MySQL driver requires no changes to any configuration but the Finish button is only enabled on the final screen. This is currently not always the case.

The only reason why "Finish" do not became enabled on the first page is not-validate connection name (which can be set only on the last page). Although DBeaver generates default connection name (based on connection settings) so potentially it is possible to enable Finish immediately.
But I'm still not sure about that..

Still thinking that a couple of extra clicks don't make much harm while showing all required options is helpful.


Equalizer is broken when an image is returning a 404

If one of the data-equalizer-watch container contains an image that's not found, equalizer isn't computing the correct height (or even doesn't set up the style="height: ??px")
Example: http://codepen.io/boertel/pen/NPwZNW (working only one page load, live editing the code seems to reflow correctly).

first section is an example where it's working because all the images are returning 200.
second section is not working because the last image is incorrect.

One workaround I found is to load a default image and play with the onerror event but it's bumpy because reflow seems to wait that the onerror is executed.
Is it possible that the issue comes from image_loaded function ? I don't see why it would affect the height, it should just ignore it.

@boertel Maybe I'm misunderstanding the usage here, but I'm showing that the current example is working correctly?

Here is how it looks on my browser (latest Chrome on Mac OS, and same with Firefox


Test fail with provider in react-native

What version of React, ReactDOM/React Native, Redux, and React Redux are you using?

React: 18.2.0
ReactDOM/React Native: 0.73.4
Redux:
React Redux:9.1.0

What is the current behavior?
When we was trying test components using jest and redux in react-native, it said "SyntaxError: Cannot use import statemen outside a module" where we used Provider from react-redux in version 9.1.0 but in react-redux version 8.1.3 it was passed in jest
What is the expected behavior?
It should pass in version 9.1.0 like other version like v 8.13
Which browser and OS are affected by this issue?
No response
Did this work in previous versions of React Redux?

[X] Yes

We updated React-Redux's packaging in v9, and it should be using ESM modules by default. Our tests show it should work with Jest. If you have a specific issue, please provide a repo that shows this error happening so we can investigate.

I created a repo : https://github.com/salarmiirzaeie/react-redux-jest/tree/main

You don't have babel-jest hooked up correctly. Add this to your jest.config.js:
"transform": {
  "\\.[jt]sx?$": "babel-jest"
},

You don't have babel-jest hooked up correctly. Add this to your jest.config.js:
"transform": {
  "\\.[jt]sx?$": "babel-jest"
},

I have the same problem. Adding this to config doesn't help

This has helped https://github.com/reduxjs/react-redux/issues/2103#issuecomment-1870389783


Support for tree-shaking in our npm packages

Our users who use Typescript are being bloated due to limitations of the commonjs module system, and tree-shaking.
If we introduce named paths for our components/icons/assets/tokens etc. we give our users an easy way to limit what code is used in their app.
Material has a good explanation for this.
Pictures from our users in regards to bundle-size:

FINISHED 🎉


Nebra Indoor Miner will not go online

Not online for 10 days. Will recognize in backend of router and pull up reports. No Peer-to -peer connections, NaN% Blockchain sync, No IP address available,

This is not a support channel.
Please email<EMAIL_ADDRESS>


Fix typo

This PR fixes a typo in a CSS code example.

See https://github.com/mrcrmn/docc/pull/14#issuecomment-613428337

This typo was fixed in 25ed42e, and should be resolved once the develop branch is merged to master.

@mrcrmn Any plans to merge those changes soon?


Pre-compiled module is missing<EMAIL_ADDRESS>

WARN Pre-compiled module is missing<EMAIL_ADDRESS>WARN Please an file issue at https://github.com/tessel/t2-cli/issues/new with this warning.
WARN
WARN This warning might be caused by one of the following:
WARN
WARN 1. A pre-compiled binary has not yet been built for this module.
WARN 2. The binary didn't compile correctly for the platform that you're developing on.
WARN    Binaries that are Linux-only or even OpenWRT-specific may cause this issue.
WARN    Try to npm install --force the affected module, then rerun your deployment command.
WARN 3. The binary may be platform specific and impossible to compile for OpenWRT.

Compilation fails on:
/bin/sh: 1: Syntax error: "(" unexpected
gyp: Call to 'LDD_VERSION="/root/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_uClibc-<IP_ADDRESS>/bin/ldd: 1: /root/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_uClibc-<IP_ADDRESS>/bin/ldd: Syntax error: "(" unexpected" node -e "require('./binding').download_vips()"' returned exit status 2 while in binding.gyp. while trying to load binding.gyp

Looks like this package uses libvips and other native dependencies -> http://sharp.dimens.io/en/stable/install/#prerequisites
I'm not sure how that compares to our OpenWRT image.


PR#312 IOSXE/show lisp eid-table vrf User ipv4 map-cache

Description
Created parser for show lisp eid-table vrf User ipv4 map-cache
Screenshots:

Checklist:

[x] I have updated the changelog.
[x] All new and existing tests passed.
[x] All new code passed compilation.

Cisco PR


close Issue #16

close Issue #16

No worries.
I have already forked an instance and compiled /checked the Sphinx code. It
is basically a documentation generator using predefined configuration /
syntax. Over the weekend, I did a bunch of tests. I can show you how things
work.
On Mon, May 9, 2022 at 8:08 AM Fernando Rios @.***>
wrote:

@.**** requested changes on this pull request.
I recommend doing a bit of research on how reStructuredText works
(homework for me too)
In source/repos.rst
https://github.com/UAL-RE/redata-docs/pull/19#discussion_r868120000:

@@ -51,7 +51,7 @@ Repositories details
+-----------------------------------------------+----------------------------+---------------+-------------------------------+-------------+---------------------------------------------+
| :ual-re:ReBACH <ReBACH>                     | N/A                        | TBC           | README__                    | main    | TBD                                         |
+-----------------------------------------------+----------------------------+---------------+-------------------------------+-------------+---------------------------------------------+
-| :ual-re:ReQUIAM <ReQUIAM>                   | |ReQUIAM_version|          | CHANGELOG__ | README__                    | master  | N/A                                         |
+| :ual-re:ReQUIAM <ReQUIAM>                   | |ReQUIAM_version|          | CHANGELOG__ | :rtd:RTD <requiam>          | master  | N/A                                         |

Won't this break the cross references? I'm not a reStructuredText expert
but I think that that the order of the anonymous hyperlink targets matters.
(i.e., the order of the references with the double underscore is now
different since README__ was deleted. This may break the rest of the __
links following this change) Please test and verify
—
Reply to this email directly, view it on GitHub
https://github.com/UAL-RE/redata-docs/pull/19#pullrequestreview-966309011,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACCPLUPQSOCOCBCJHBA2AV3VJES75ANCNFSM5VJOKWXA
.
You are receiving this because you authored the thread.Message ID:
@.***>

I've re-opened the issue #16 as the changes seem to have broken the references


Secret without bearer and locations should be urls

Motivation
I discovered after I deployed that token shouldn't have Bearer in it, because we add that in the app-config.yaml. We also need to make sure that each location is a URL.
Approach

Removed Bearer prefix2.
Changed location type

📣 NOTIFICATION
You are receiving this message because we did not publish any packages.
Generated by @thefrontside/actions


Swap out UserPasswordEncoderInterface with UserPasswordHasherInterface

Swap out UserPasswordEncoderInterface (deprecated in 5.3, removed in 6.0) with UserPasswordHasherInterface, resolves #1

Tested this in a scratch project against 5.3.0-RC1 with security.yaml's encoders option changed to password_hashers.


Crash in some pointer

Check List
Thanks for considering to open an issue. Before you submit your issue, please confirm these boxes are checked.

[x] I have read the wiki page and cheat sheet, but there is no information I need.
[x] I have searched in existing issues, but did not find a same one.
[x] I want to report a problem instead of asking a question. It'd better to use kingfisher tag in Stack Overflow to ask a question.

Issue Description
What
We keep receiving some crash after upgrade to 6.0.0. The latest version 6.2.1 has the issue, too.
Here is the trace:
0  libobjc.A.dylib         _objc_retain
1  Kingfisher              0x1dac4 (-0x101704000 +<PHONE_NUMBER>)
2  Kingfisher              0x5a644 (-0x101704000 +<PHONE_NUMBER>)
3  Kingfisher              0x823b8 (-0x101704000 +<PHONE_NUMBER>)
4  Kingfisher              0x8d490 (-0x101704000 +<PHONE_NUMBER>)
5  Kingfisher              0x8d584 (-0x101704000 +<PHONE_NUMBER>)
6  CFNetwork               ___67-[NSURLSession delegate_dataTask:didReceiveData:completionHandler:]_block_invoke.264
7  Foundation              ___NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__
8  Foundation              -[NSBlockOperation main]
9  Foundation              -[__NSOperationInternal _start:]
10 libdispatch.dylib       __dispatch_client_callout
11 libdispatch.dylib       __dispatch_block_invoke_direct$VARIANT$mp
12 libdispatch.dylib       __dispatch_client_callout
13 libdispatch.dylib       __dispatch_block_invoke_direct$VARIANT$mp
14 libdispatch.dylib       _dispatch_block_perform$VARIANT$mp
15 Foundation              ___NSOQSchedule_f
16 libdispatch.dylib       __dispatch_client_callout
17 libdispatch.dylib       __dispatch_continuation_pop$VARIANT$mp
18 libdispatch.dylib       __dispatch_async_redirect_invoke$VARIANT$mp
19 libdispatch.dylib       __dispatch_root_queue_drain
20 libdispatch.dylib       __dispatch_worker_thread3
21 libsystem_pthread.dylib __pthread_wqthread

Do you have any suggestion?
Thanks!

New information is the crash only happened in iOS 11 and iOS 12


[Model] Support SigLIP encoder and alternative decoders for LLaVA models

Based on #7067, this PR partially addresses #7143 by enabling SigLIP to be used in LLaVA models. The same code also enables them to load other LLM decoders.
This PR also fixes two hidden bugs in the calculation for the number of placeholder tokens:

For CLIP encoder, the feature size should be increased by 1 because the CLS token is prepended to the output.
For LLaVA models, the feature size should be reduced by 1 when vision_feature_select_strategy="default".

These two miscalculations cancelled out each other prior to this PR. They were discovered when trying to support SigLIP in #7149 because SigLIP encoder, unlike CLIP encoder, does not output the CLS token, resulting in an off-by-one error when filling in placeholders.
FIX #7149

I have tried this method 3 hour ago. Bue the error as follos:
[rank0]:   File "/data/vayu/train/workspace/vllm/vllm/model_executor/models/utils.py", line 29, in merge_vision_embeddings
[rank0]:     raise ValueError(
[rank0]: ValueError: Attempted to assign 11 x 728 = 8008 image tokens to 8019 placeholders
It's seems that the model need more image tokens.

I have tried this method 3 hour ago. Bue the error as follos: [rank0]: File "/data/vayu/train/workspace/vllm/vllm/model_executor/models/utils.py", line 29, in merge_vision_embeddings [rank0]: raise ValueError( [rank0]: ValueError: Attempted to assign 11 x 728 = 8008 image tokens to 8019 placeholders It's seems that the model need more image tokens.

Can you show the input to your model? Also note that we don't support multiple image input per prompt for now.

OK I have found an issue with the image feature size calculation. Fixing...

Fixed, can you try again?

I will try it maybe reply tonight or tomorrow.
发自我的iPhone
------------------ Original ------------------
From: Cyrus Leung @.>
Date: Mon, Aug 5, 2024 6:44 PM
To: vllm-project/vllm @.>
Cc: Brench @.>, Comment @.>
Subject: Re: [vllm-project/vllm] [Model] Support alternative LLM backbone andSigLIP encoder for LLaVA model (PR #7153)
Fixed, can you try again?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

can you try this python file as follows?
'
import requests
from PIL import Image
from vllm import LLM, SamplingParams
def run_llava_next():
llm = LLM(model="TIGER-Lab/Mantis-8B-siglip-llama3", max_model_len=4096)
prompt = " <image>\nWhat is shown in this image? "
url = "https://h2o-release.s3.amazonaws.com/h2ogpt/bigben.jpg"
image = Image.open(BytesIO(requests.get(url).content))
sampling_params = SamplingParams(temperature=0.8,
                                 top_p=0.95,
                                 max_tokens=100)

outputs = llm.generate(
    {
        "prompt": prompt,
        "multi_modal_data": {
            "image": image
        }
    },
    sampling_params=sampling_params)

generated_text = ""
for o in outputs:
    generated_text += o.outputs[0].text

print(f"LLM output:{generated_text}")

if name == "main":
run_llava_next()
`
because my environment has some problems : '_OpNamespace' '_C' object has no attribute 'rms_norm'

import requests from PIL import Image from vllm import LLM, SamplingParams def run_llava_next():     llm = LLM(model="TIGER-Lab/Mantis-8B-siglip-llama3", max_model_len=4096)     prompt = " <image>\nWhat is shown in this image? "     url = "https://h2o-release.s3.amazonaws.com/h2ogpt/bigben.jpg"     image = Image.open(BytesIO(requests.get(url).content))     sampling_params = SamplingParams(temperature=0.8,                                      top_p=0.95,                                      max_tokens=100)     outputs = llm.generate(         {             "prompt": prompt,             "multi_modal_data": {                 "image": image             }         },         sampling_params=sampling_params)     generated_text = ""     for o in outputs:         generated_text += o.outputs[0].text     print(f"LLM output:{generated_text}") if __name__ == "__main__":     run_llava_next()
can you try this? because my conda env has a problem:
[rank0]: AttributeError: '_OpNamespace' '_C' object has no attribute 'rms_norm'

from io import BytesIO
import requests
from PIL import Image
from vllm import LLM, SamplingParams
def run_llava_next():
llm = LLM(model="TIGER-Lab/Mantis-8B-siglip-llama3", max_model_len=4096)
prompt = " <image>\nWhat is shown in this image? "
url = "https://h2o-release.s3.amazonaws.com/h2ogpt/bigben.jpg"
image = Image.open(BytesIO(requests.get(url).content))
sampling_params = SamplingParams(temperature=0.8,
                                 top_p=0.95,
                                 max_tokens=100)

outputs = llm.generate(
    {
        "prompt": prompt,
        "multi_modal_data": {
            "image": image
        }
    },
    sampling_params=sampling_params)

generated_text = ""
for o in outputs:
    generated_text += o.outputs[0].text

print(f"LLM output:{generated_text}")

if name == "main":
run_llava_next()

You should delete any compiled libraries from your local vLLM repo, and reinstall vLLM.

Based on the test results, the model can run in vLLM but the output is gibberish (probably something wrong with the input processor). Since this PR isn't aimed at fully implementing Mantis, we can create another PR after this one to resolve those issues.

I know the mistake. I need to build on CUDA. Your branch is base on AMD. LOL

Thanks for your help!!!

Yep,because there are some modifications of the 'LlavaForConditionalGeneration' in Mantis about the tokens. So Mantis depends on its own class to use.

may be should use some function to remove the unexpected tag.
reference:https://github.com/TIGER-AI-Lab/Mantis/blob/main/mantis/models/mllava/utils.py
import PIL
import torch
from .modeling_llava import LlavaForConditionalGeneration
from .processing_llava import MLlavaProcessor
# from ..conversation import conv_mllava_v1_mmtag as default_conv
from ..conversation import conv_mllava_v1 as default_conv, conv_templates

from typing import List, Tuple, Union, Tuple

def chat_mllava(
    text:str,
    images: List[Union[PIL.Image.Image, str]],
    model:LlavaForConditionalGeneration,
    processor:MLlavaProcessor,
    max_input_length:int=None,
    history:List[dict]=None,
    **kwargs) -> Tuple[str, List[dict]]:
    """
    Chat with the Mllava model
    Args:
        text: str, the text to be sent to the model, where <image> will be the placeholder for the image
        images: List[PIL.Image.Image], the images to be sent to the model, or None
        model: LlavaForConditionalGeneration, the model to be used
        processor: MLlavaProcessor, the processor to be used
        max_input_length: int, the maximum input length
        history: List[dict], list of messages in the conversation as history. Each message is a dictionary {"role": "ASSISTANT/USER", "text": "the message"}. If None, the conversation will start from scratch
        kwargs: dict, the generation kwargs
    Returns:
        Tuple[str, List[dict]], the generated text and the history of the conversation

    """
    if "llama-3" in model.language_model.name_or_path.lower():
        conv = conv_templates['llama_3']
        terminators = [
            processor.tokenizer.eos_token_id,
            processor.tokenizer.convert_tokens_to_ids("<|eot_id|>")
        ]
    else:
        conv = default_conv
        terminators = None
    kwargs["eos_token_id"] = terminators
    conv = conv.copy()
    conv.messages = []
    if history is not None:
        for message in history:
            assert message["role"] in conv.roles
            conv.append_message(message["role"], message["text"])
        if text:
            assert conv.messages[-1][0] == conv.roles[1], "The last message in the history should be the assistant, if the given text is not empty"
            conv.append_message(conv.roles[0], text)
            conv.append_message(conv.roles[1], "")
            history.append({"role": conv.roles[0], "text": text})
            history.append({"role": conv.roles[1], "text": ""})
        else:
            if conv.messages[-1][0] == conv.roles[1]:
                assert conv.messages[-1][1] == "", "No user message should be provided"
            else:
                assert conv.messages[-1][0] == conv.roles[0], "The last message in the history should be the user, if the given text is empty"
                conv.append_message(conv.roles[0], "")
                history.append({"role": conv.roles[0], "text": ""})
    else:
        history = []
        history.append({"role": conv.roles[0], "text": text})
        history.append({"role": conv.roles[1], "text": ""})
        conv.append_message(conv.roles[0], text)
        conv.append_message(conv.roles[1], "")
    assert conv.messages[-1][0] == conv.roles[1] and conv.messages[-1][1] == "", "Format check"
    assert history[-1]["role"] == conv.roles[1] and history[-1]["text"] == "", "Format check"

    prompt = conv.get_prompt()
    if images:
        for i in range(len(images)):
            if isinstance(images[i], str):
                images[i] = PIL.Image.open(images[i]).convert("RGB")

    inputs = processor(images=images, text=prompt, return_tensors="pt", truncation=True, max_length=max_input_length)
    for k, v in inputs.items():
        if v is not None:
            if isinstance(v, torch.Tensor):
                inputs[k] = v.to(model.device)
            elif isinstance(v, list):
                inputs[k] = [x.to(model.device) for x in v]
            else:
                raise ValueError(f"Invalid input type: {type(v)}")

    output_ids = model.generate(**inputs, **kwargs)
    output_ids = output_ids[0]

    # remove the input tokens
    generated_ids = output_ids[inputs["input_ids"].shape[-1]:]
    generated_text = processor.decode(generated_ids, skip_special_tokens=True)

    history[-1]["text"] = generated_text

    return generated_text, history

reference:
import torch
from PIL import Image
from abc import abstractproperty
from .base import BaseModel
from ..smp import *
from ..dataset import DATASET_TYPE
import warnings

class Mantis(BaseModel):
    """
    Mantis Model
    This implementation is adpated from the Llava model from llava.py and the Idefics model from idefics.py
    """
    INSTALL_REQ = True
    INTERLEAVE = True

    DEFAULT_IMAGE_TOKEN = '<image>'
    IMAGE_TOKEN_INDEX = -200

    def __init__(self, model_path='TIGER-Lab/Mantis-8B-siglip-llama3', **kwargs):
        assert model_path is not None
        try:
            from mantis.models.mllava import LlavaForConditionalGeneration, MLlavaProcessor
            from mantis.models.mfuyu import MFuyuForCausalLM, MFuyuProcessor
            from mantis.models.conversation import conv_mllava_v1 as default_conv, conv_templates
        except:
            warnings.warn(
                "Mantis is not installed. Please install Mantis to use this model.Please use 'pip install "
                "git+https://github.com/TIGER-AI-Lab/Mantis.git' to install"
            )

        try:
            from transformers import AutoModelForVision2Seq, AutoProcessor
        except Exception as e:
            warnings.warn("Upgrade transformers to use Mantis's idefics model.\nError: %s" % e)
        except:
            warnings.warn('Please `pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git')

        # inference implementation for attention, can be "sdpa", "eager", "flash_attention_2".
        # Seems FA2 is not effective during inference:
        # https://discuss.huggingface.co/t/flash-attention-has-no-effect-on-inference/73453/5
        # if is_flash_attn_2_available:
        #     best_fit_attn_implementation = "flash_attention_2"
        # flash_attn has a bug that says: ERROR Error query and key must have the same dtype in generating

        try:
            import flash_attn
            best_fit_attn_implementation = 'flash_attention_2'
        except ImportError:
            best_fit_attn_implementation = 'eager'
        self.model_path = model_path
        attn_implementation = best_fit_attn_implementation
        self._is_idefics = 'idefics' in model_path.lower()
        # Here load the "non-idefics" Mantis model.
        if not self._is_idefics:
            if 'fuyu' in model_path.lower():
                self.processor = MFuyuProcessor.from_pretrained(self.model_path)
                model = MFuyuForCausalLM.from_pretrained(
                    self.model_path,
                    device_map='cuda',
                    attn_implementation=attn_implementation,
                    torch_dtype=torch.float16
                )
            else:
                self.processor = MLlavaProcessor.from_pretrained(self.model_path)
                model = LlavaForConditionalGeneration.from_pretrained(
                    self.model_path,
                    device_map='cuda',
                    attn_implementation=attn_implementation,
                    torch_dtype=torch.float16
                )
        else:
            self.processor = AutoProcessor.from_pretrained(self.model_path)
            model = AutoModelForVision2Seq.from_pretrained(
                self.model_path,
                device_map='cuda',
                torch_dtype=torch.float16
            )

        model = model.eval()
        self.model = model.cuda()
        kwargs_default = dict(do_sample=False, temperature=0, max_new_tokens=1024, top_p=None, num_beams=1)
        kwargs_default.update(kwargs)
        self.kwargs = kwargs_default
        warnings.warn(f'Following kwargs received: {self.kwargs}, will use as generation config. ')

        self.tokenizer = self.processor.tokenizer
        self.default_conv = default_conv
        self.conv_templates = conv_templates

    def use_custom_prompt(self, dataset):
        assert dataset is not None
        if DATASET_TYPE(dataset) == 'MCQ':
            return True
        return False

    def build_prompt(self, line, dataset=None):
        assert self.use_custom_prompt(dataset)
        assert dataset is None or isinstance(dataset, str)
        tgt_path = self.dump_image(line, dataset)

        question = line['question']
        hint = line['hint'] if ('hint' in line and not pd.isna(line['hint'])) else None
        if hint is not None:
            question = hint + '\n' + question
        options = {
            cand: line[cand]
            for cand in string.ascii_uppercase
            if cand in line and not pd.isna(line[cand])
        }
        for key, item in options.items():
            question += f'\n{key}. {item}'
        prompt = question

        if len(options):
            prompt += (
                '\n请直接回答选项字母。' if cn_string(prompt) else
                "\nAnswer with the option's letter from the given choices directly."
            )
        else:
            prompt += '\n请直接回答问题。' if cn_string(prompt) else '\nAnswer the question directly.'
        message = [dict(type='image', value=s) for s in tgt_path]
        message.append(dict(type='text', value=prompt))
        return message

    def output_process(self, answer):
        if '<s>' in answer:
            answer = answer.replace('<s>', '').strip()
        if '[/INST]' in answer:
            answer = answer.split('[/INST]')[1].strip()
        elif 'ASSISTANT:' in answer:
            answer = answer.split('ASSISTANT:')[1].strip()
        elif 'assistant\n' in answer:
            answer = answer.split('assistant\n')[1].strip()
        elif '<|end_header_id|>\n\n' in answer:
            answer = answer.split('<|end_header_id|>\n\n')[2].strip()

        if '</s>' in answer:
            answer = answer.split('</s>')[0].strip()
        elif '<|im_end|>' in answer:
            answer = answer.split('<|im_end|>')[0].strip()
        elif '<|eot_id|>' in answer:
            answer = answer.split('<|eot_id|>')[0].strip()
        elif '<end_of_utterance>':
            answer = answer.split('<end_of_utterance>')[0].strip()
        return answer

    def generate_inner(self, message, dataset=None):
        content, images = '', []
        ide_content, question = [], ''
        for msg in message:
            if msg['type'] == 'text':
                content += msg['value']
                question += msg['value']
            else:
                images.append(Image.open(msg['value']).convert('RGB'))
                content += (self.DEFAULT_IMAGE_TOKEN + '\n')
                ide_content.append({'type': 'image'})
        if self._is_idefics:
            # Follow the idefics implementation:
            ide_content.append({'type': 'text', 'text': question})
            prompt = [{'role': 'user', 'content': ide_content}]
            prompt = self.processor.apply_chat_template(prompt, add_generation_prompt=True)
        else:
            # Follow the Mantis code base to make sure they are consistent:
            # https://github.com/TIGER-AI-Lab/Mantis/blob/main/mantis/models/mllava/utils.py#L33
            # Users don't need to define chat template as it is done here
            if 'llama-3' in self.model.language_model.name_or_path.lower():
                conv = self.conv_templates['llama_3']
                terminators = [
                    self.processor.tokenizer.eos_token_id,
                    self.processor.tokenizer.convert_tokens_to_ids('<|eot_id|>')
                ]
            else:
                conv = self.default_conv
                terminators = None

            # Using EOT because end of *text* is more accurate for what we're doing than end of *sentence*
            if 'eos_token_id' not in self.kwargs:
                self.kwargs['eos_token_id'] = terminators

            conv = conv.copy()
            conv.append_message(conv.roles[0], content)
            conv.append_message(conv.roles[1], '')
            assert conv.messages[-1][0] == conv.roles[1] and conv.messages[-1][1] == '', 'Format check'
            prompt = conv.get_prompt()

        inputs = self.processor(prompt, images, return_tensors='pt', truncation=True)
        # FIXME: Fuyu model would return a list instead of a pytorch tensor. This weird behavior needs fixing.
        if 'image_patches' in inputs.keys():
            inputs['image_patches'] = inputs['image_patches'][0]
        inputs = {k: v.to(self.model.device) for k, v in inputs.items()}
        output = self.model.generate(**inputs, **self.kwargs)
        output = output[0]
        generated_ids = output[inputs['input_ids'].shape[-1]:]
        answer = self.processor.decode(generated_ids, skip_special_token=True)
        answer = self.output_process(answer)
        return answer


ENH: upgrade external library nmslib to latest

Issue #, if available:
#128
Description of changes:
This PR upgrades the nmslib to latest and change the impacted file/folder names, import paths, APIs and security settings.
Note:

I verified backward compatibility with the examples on README.md, where I stored indices and docs in V1736 and retrieve/search in V206. It is possible that there will be corner cases missed and we should be prepared to roll back. Not sure if it is necessary to formulate a test suite on backward compatibility.
It is still cumbersome to update everything impacted by the nmslib version in the codebase. If it is going to be done on a regular basis, we should probably improve the infrastructure.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hi @chenqi0805 , thanks for working on this.
I think that backwards compatibility is very important. I think the following cases need to be tested to confirm that they do not impact performance:

Document set is indexed with v<IP_ADDRESS>, and then loaded/searched with v2.0.6.
Document set is index with v<IP_ADDRESS>, and then more documents are added to the index with v2.0.6. Then, search/merge operations are tested.

I can work on testing these.
@vamshin and @chenqi0805 , if you can think of any other test cases, please add them here.
Additionally, with respect to library naming, I believe that NMSLIB version was added to our naming conventions in case backwards compatibility was broken. However, if the library versions are backwards compatible, I see no reason to include NMSLIB version in our naming conventions. This possibly could be changed in another PR.

@jmazanec15 Thanks a lot for running and elaborating the backward compatibility test! Just curious about how you monitored performance degradation, i.e. what metrics did you inspect maybe on cloud watch dashboard? I hope to carry out similar tests w.r.t dot product space #161 .
@vamshin If you do not have further comments, I will merge the PR.

@jmazanec15 Thanks a lot for running and elaborating the backward compatibility test! Just curious about how you monitored performance degradation, i.e. what metrics did you inspect maybe on cloud watch dashboard? I hope to carry out similar tests w.r.t dot product space #161 .
@vamshin If you do not have further comments, I will merge the PR.

@chenqi0805 will take a quick look today and get back on this. So far looks good from Jack's experiments

Hi @chenqi0805, I have a few recall tests I run and ES Rally scripts I used based off ANN-Benchmark datasets. Eventually, I will submit PR and add them to the k-NN repo.

@jmazanec15 Thanks a lot for running and elaborating the backward compatibility test! Just curious about how you monitored performance degradation, i.e. what metrics did you inspect maybe on cloud watch dashboard? I hope to carry out similar tests w.r.t dot product space #161 .
@vamshin If you do not have further comments, I will merge the PR.

@chenqi0805 will take a quick look today and get back on this. So far looks good from Jack's experiments

@chenqi0805 Done. Thanks for the work. Feel free to merge the PR. @jmazanec15 thanks for thorough testing.


[ntop] add support for arrays and IPv6

Added ntop support for array types of sizes 4 and 16
Added ntop support for IPv6
Optimized number of instructions generated for ntop and memory used, both in the BPF stack and maps
Make first argument of ntop optional, inferring the af_type from the address type
Updated tools to handle IPv6 and use AF_* macros

Making it option is definitely preferred :+1: it  was only required before because it needed to be explicit that only AF_INET was supported at the time.

Updated with:

Make first parameter of ntop optional and infer the AF type from the type and size of our address paramenter.
Some optimizations:

Only allocate what we need on the stack (8 bytes for IPv4, 20 bytes for IPv6)
Avoid copying things back and forth on the stack, LLVM doesn't seem smart enough to optimize this.

I'll update docs and tests now.

For literal integers, ntop will be reduced to essentially two operations:
#: (18) r1 = 0x[address][af_type]
#: (7b) *(u64 *)(r10 -x) = r1

For example:
$ sudo ./src/bpftrace -ve 'BEGIN { printf("%s\n", ntop(0x0100007f));}'
Attaching 1 probe...

Program ID: 405

Bytecode:
0: (bf) r6 = r1
1: (18) r1 = 0x100007f00000002
3: (7b) *(u64 *)(r10 -8) = r1
4: (b7) r1 = 0
5: (7b) *(u64 *)(r10 -16) = r1
6: (18) r7 = 0xffff8b6d7e5f3100
8: (85) call bpf_get_smp_processor_id#8
9: (bf) r4 = r10
10: (07) r4 += -16
11: (bf) r1 = r6
12: (bf) r2 = r7
13: (bf) r3 = r0
14: (b7) r5 = 16
15: (85) call bpf_perf_event_output#25
16: (b7) r0 = 0
17: (95) exit
processed 16 insns, stack depth 16

Attaching BEGIN
Running...
<IP_ADDRESS>
^C

Not particularly useful, because we'll fetch the address using probe_read on most cases. But I'm happy with this result :)

thanks!

This patch has broken some stuff related to reading in structs.
e.g. if you try running the test in tests/codegen/struct_save_nested.cpp on a real program, it will refuse to load now.

Fixed in #579


Increment version for cognitiveservices releases

Increment package version after release of com.azure.resourcemanager azure-resourcemanager-cognitiveservices

/check-enforcer override


Add goofys as user agent for s3 request

This PR add "goofys/version-hash" to the user agent list for each request. You will be able to identify which requests come from goofys and which version if you enable server access logging or CloudTrail on your buckets.
example for debug_s3 log
2022/05/18 10:08:32.838728 s3.DEBUG DEBUG: Request s3/ListObjectsV2 Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /<REDACTED>?delimiter=%2F&list-type=2&prefix= HTTP/1.1
Host: s3.eu-west-1.amazonaws.com
User-Agent: goofys/0.24.0-829d8e5ce20faa3f9f6f054077a14325e00e9249 aws-sdk-go/1.38.7 (go1.18.2; linux; amd64)
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=<REDACTED>, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<REDACTED>
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220518T100832Z

example for server access log
fda01774e8671afabd99d08ccfcb4ef3a0fde3b84c04977a33cd49c863ac0b3e bucket-name [18/May/2022:10:07:30 +0000] <REDACTED> arn:aws:iam::<REDACTED>:user/<REDACTED> CVTB43X4F1MAXG2X REST.GET.OBJECT largefile "GET /bucket-name/largefile HTTP/1.1"<PHONE_NUMBER>0<PHONE_NUMBER>0 685 128 "-" "goofys/0.24.0-829d8e5ce20faa3f9f6f054077a14325e00e9249 aws-sdk-go/1.38.7 (go1.18.2; linux; amd64)" - <REDACTED> SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.eu-west-1.amazonaws.com TLSv1.2 -

@kahing would you be able to take a look?
I also think it might be a good opportunity to release a new version after this is merged, it's been a while since the last version and things are starting to break.
I'm happy to help if there are any tests required before the release.


Class UpdateHelper\ComposerPlugin contains 2 abstract methods and must therefore be declared abstract or implement the remaining methods

Fatal error: Class UpdateHelper\ComposerPlugin contains 2 abstract methods and must therefore be declared abstract or implement the remaining methods (Composer\Plugin\PluginInterface::deactivate, Composer\Plugin\PluginInterface::uninstall) in /home/teamcity/2017.2.3/buildAgent/work/942ad3a15f527ecb/vendor/kylekatarnls/update-helper/src/UpdateHelper/ComposerPlugin.php on line 11

Please provide:

[ ] The code/command you run
[ ] The result you expected from this code/command
[ ] Your composer.json
[ ] The PHP version you run

composer install -vvv
install packages

$php -v
PHP 7.1.33 (cli) (built: Mar 17 2020 09:14:41) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies

php 7.1.33

Hi, may also know your version of composer?
I will dig deeper, but for now I can yet see a blocking point: you use laravelrus/localized-carbon
This package actually breaks Carbon as Carbon yet support localization for a long time (since 1.18) and has way more feature and languages available than localized-carbon including of course diffForHumans(), and this package also prevent you from upgrading to Carbon 2 and so to modern Laravel versions.
I will try to make this package compatible with. But even if I do, you may be warned you have many no-longer-supproted or obsolete packages.
I would highly recommend to remove the package laravelrus/localized-carbon (which makes no point), then a good next step would be to upgrade Laravel to at least 5.8.

$ composer --version
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Composer version 1.4.1 2017-03-10 09:29:45
Disabling the carbon localization package did not help.
Unfortunately, there is no way to upgrade the project to version 5.8 or higher.

Updated composer.
$ composer --version
Do not run Composer as root / super user! See https://getcomposer.org/root for details
Composer version 1.10.1 2020-03-13 20:34:27
The result is the same.

I did not say removing laravelrus/localized-carbon would fix this error, I said it's useless as Carbon already has localization, so it's blocking features provided by default and it prevents to update Laravel.
And Composer 1.4.1 or 1.10.1 does not contain deactivate and uninstall methods in PackageInterface as you can see:
https://github.com/composer/composer/blob/1.4.1/src/Composer/Package/PackageInterface.php
So I don't know where those Composer\Plugin\PluginInterface methods come from. Note than this package has been installed properly thousand times so it looks like a package is overriding Composer stuff and this sounds bad.

There is a suspicion that the problem is in the TeamCity service.

To add to this conversation, the company I work for have started seeing this same error in our GitHub Actions this week.

Clearly not due to a change in the update-helper package as we did not release any change of this package since July.

Suspicions came true. ТeamСity by default started using composer 2.0-dev. For this reason, the assembly of the project broke down.

Same, one of our phpunit setup scripts downloads https://getcomposer.org/composer.phar which has started returning 2.0-dev.

I was facing the same issue and the reason was using composer 2, downgrading it to version 1 worked fine for me.

Once you will update your other dependencies you will be able to safely upgrade again to composer 2 (and you should). The whole point of this library is to help you to stay up to date and safe.
Taking upgrade paths will save your time on the long run while taking downgrade paths will bring you more troubles soon again.

I had nothing modified in my vendor folder, and reinstalling packages wouldn't be a problem for me, so my solution was this:
rm -rf vendor
rm -f composer.lock
composer install

worked for me.
Use at your own risk. I'm not responsible  for anything :) Just showing you another way to overcome this.

rm composer.lock
composer install

Worked for me Like charm

composer require kylekatarnls/update-helper:"^1.2.1"

i use Composer version 2.0.8 2020-12-03 17:20:38 is error
i back composer version 1.8.6 pass

This has been mentioned before
Please also read this note

composer self-update -r

rm -rf vendor/kylekatarnls/update-helper
composer update kylekatarnls/update-helper

this worked for with composer v2

@ankurk91 You did well man!

rm -rf vendor
composer require kylekatarnls/update-helper:"^1.2.1"

That way you don't loose all your current versions in the lock file

This helped me... Thanks

Just came by this problem while working on an old project.
composer update
and everything is looking great


Mostly safe refactorings of CloudDebugHistoricalSnapshots

@patflynn This cl performs a number of relatively safe, automatic refactorings in the CloudDebugHistoricalSnapshots class and a few related classes. It also expands on the intenral documentation in the comments. The main refactoring that was performed was pulling out a lot of anonymous inner classes into non-anonymous inner classes.
Step 2 is going to be to further separate the model from the GUI so we can better test this code.

@joaoandremartins this should probably go in before work on several outstanding bugs since it cleans up a lot of the code where the bugs seemj to be hiding.

I'\m looking at this next. Should have comments this afternoon.
On Mon, Dec 14, 2015 at 9:41 AM, Elliotte Rusty Harold <
<EMAIL_ADDRESS>wrote:

@joaoandremartins https://github.com/joaoandremartins this should
probably go in before work on several outstanding bugs since it cleans up a
lot of the code where the bugs seemj to be hiding.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-intellij/pull/310#issuecomment-164454153
.

Huge improvement. Thanks.
LGTM


Update changelog ahead of 0.20.0 final

#3522 wasn't included in 0.20.0rc2, it will be included in 0.20.0 (final)

Ah... it looks like GitHub Actions respects [skip ci] for push and pull_request triggers, but perhaps not for pull_request_target?


test: extend prometheus/alertmanager e2e test suites

To run these e2e tests you need minikube installed on your local machine. In addition to that two recently added APIs are required of upstream Prometheus (targets/alertmanager API). Therefore to run these tests your cluster needs to use an image that already has those. As there is no release out yet that does the master image is used here.

make sure you set your DOCKER_HOST to the minikube provided one

Then run these e2e tests with (this also builds the operator from current working tree tagged with the short HEAD commit hash):
make e2e

Prometheus test suite now consists of:

create/delete
scale up/down
version migration
config generation, target discovery and basic query
alertmanager discovery

Alertmanager test suite now consists of:

create/delete
scale up/down
version migration

The Alertmanager version migration test is currently commented out, as it is failing, I am still figuring out whether this is a bug in the operator or a problem with the test, but it seems that it is a bug that needs further investigation. However, before spending lots of time on fixing the existing implementation the Alertmanager operator part should be ported to use the same event system as the Prometheus part. There was a bug in the way the Alertmanager version sync worked, I used that chance and implemented the PrometheusStatus equivalent for the Alertmanager.
Another issue, however, this may be in upstream Prometheus is with the Alertmanager discovery test, that is very flaky when removing the time.Sleep(time.minute), before creating an Alertmanager cluster after having created the Prometheus instance. The flaky case seem to be that the discovery mechanism only starts to pick up the Alertmanager cluster after the first cache resyncPeriod. This needs to be investigated upstream.
I am submitting this for a preliminary code review, as I would like some feedback while I dig into the above mentioned issues.
@fabxc @alexsomesan
It might be necessary to run tests with a different size than standard minikube, I have not extensively tested that. I start minikube with the following settings:
minikube start --vm-driver="virtualbox" --kubernetes-version="v1.5.1" --memory 4096 --cpus 2

The Alertmanager version update test is timing out in my setup but all the other ones are passing.
Looks good in general.
Okay to merge with me. The failing one might be a flake we can fix later.

Alright. Then let's move forward with this.

🚢 💯


Optimize check index problems

Optimize nonce retrieval
Print block number where the first issue was found

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 14 (0.0%)  changed or added relevant lines in 1 file are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.08%) to 87.793%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

safe_transaction_service/history/management/commands/check_index_problems.py
0
14
0.0%

Files with Coverage Reduction
New Missed Lines
%

safe_transaction_service/history/management/commands/check_index_problems.py
2
0%

Totals

Change from base Build<PHONE_NUMBER>:
0.08%

Covered Lines:
5912

Relevant Lines:
6734

💛  - Coveralls


Support Batch Download Youtube Videos

current /dl yotube url can only download a video,
can support /dl playlist url, or /dl channel url or /dl muti youtube url to download muti videos ?

current /dl yotube url can only download a video, can support /dl playlist url, or /dl channel url or /dl muti youtube url to download muti videos ?

I will


Add keymanager interface

Closes #255

Btw this seems to have broke the oasis-core-commit-ref action.
Since the REF is now:
https://github.com/oasisprotocol/oasis-sdk/blob/5e48b25fba78335613b5b472f62497138685de5c/.github/actions/oasis-core-commit-ref/action.yml#L13-L14
echo $REF
a1c9fe982c32d550e8eac530bf8e79402728d64a
a509afa

Probably cause there are now multiple oasis-core dependencies in Cargo.lock

Looks like I forgot to change the dependency for the simple-keyvalue runtime when rebasing. #403 should fix that.


Question about attention-based GRN discovery

Hi, I am trying to reproduce the Attention-based GRN discovery workflow for perturbation data in Fig.6a, but I cannot finish it because of some unknown detail information, such as the implementation of rank-normalization. Could u pls give me some hints or update the tutorial  about it.

Sure, we will add the tutorial asap. The update is going to be ready in a few days.


Add serial sensor

Description:
This sensor platform allows one to read data directly for a serial port.
Details to build a sensor are covered in home-assistant/home-assistant.github.io#3620.
Pull request in home-assistant.github.io with documentation (if applicable): home-assistant/home-assistant.github.io#3619
Example entry for configuration.yaml (if applicable):
sensor:
  - platform: serial
    port: /dev/ttyACM0

Checklist:
If user exposed functionality or configuration variables are added/changed:

[x] Documentation added/updated in home-assistant.github.io

If the code communicates with devices, web services, or third-party tools:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[x] New dependencies have been added to the REQUIREMENTS variable ([example][ex-requir]).
[x] New dependencies are only imported inside functions that use them ([example][ex-import]).
[x] New dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[x] New files were added to .coveragerc.

This looks great! but, could be able to work with a buffer of multiple lines? And make num_lines: 1 as default.
Perhaps working with a deque and joining the state when accessed?
self._state = deque([], max_lengh=config.get(CONF_NUMBER_LINES))
...

self._state.append(line.decode('utf-8').strip())
self.async_schedule_update_ha_state()

...

@property
def state(self):
    return '\n'.join(self._state)

What do you think?

I kept this first approach very minimal. There is no template support and no error checking for the incoming data.
I would prefer if we wait with adding additional features till it's proven that the platform works with different device not only my own. Also, pyserial-asyncio is fairly new.

Ok, I see.
I'll give it a try, I saw your blog post and I understand the objective.
I work a lot with microcontrollers and I thought this could be good to control the logging of these devices when developing... that's the reason for asking for multiline support.
We'll wait, then (and test this as is it now to provide feedback if any problem shows up)

Perhaps I shouldn't have looked at the imap sensor. From there it felt like not to watch out for EVENT_HOMEASSISTANT_START would be the way to go.


Apply logger level

With this commit we fix a bug that custom logger levels were always set to INFO no matter what was there in settings.yaml. Specifically, I've been trying to set DEBUG level for $vcloud_log logger but even if server confirmed following settings when started:
...
:log:
  :level_aws: info
  :level_azure: warn
  :level_datawarehouse: info
  :level_kube: info
  :level_rhevm: info
  :level_vcloud: debug  <----------------------- OK
...

the logger wasn't printing DEBUG messages, only INFO. Not sure why this happens, but in apply_config_value() the "old_level" was reported to be 0 (debug) already when it was actually 1 (info). Quick fix is simple: always set log level no matter what logger reports as current value.
Am I perhaps missing something?
@miq-bot assign @agrare
@miq-bot add_label bug
/cc @gberginc

@miha-plesko Do you have a script to reproduce this problem?  Everything is working fine for me in rails console.
irb(main):001:0> $vcloud_log.level
=> 1
irb(main):002:0> $vcloud_log.loggers.collect(&:level)
=> [1]
irb(main):003:0> $vcloud_log.level = 0
=> 0
irb(main):004:0> $vcloud_log.level
=> 0
irb(main):005:0> $vcloud_log.loggers.collect(&:level)
=> [0]
irb(main):006:0> $vcloud_log.level = 1
=> 1
irb(main):007:0> $vcloud_log.level
=> 1
irb(main):008:0> $vcloud_log.loggers.collect(&:level)
=> [1]

Hi @bdunne,
I never said setting logger level doesn't work when logger.level = N is invoked. Problem is that this line didn't get invoked for some reason 😄
However, this issue has been resolved 16 days ago with another PR, so I'm closing the issue.


[DISCUSS] AutoCloseable

I think that some classes, like: CassandraDocumentEntityManager should implement AutoCloseable https://docs.oracle.com/javase/8/docs/api/java/lang/AutoCloseable.html.
WDYT? :)

@danielsoro is already done, please, look to ColumnFamilyManager, the interface that CassandraDocumentEntityManager implements. It's already implements AutoCloseable

@otaviojava Yeh.. I can see here: https://github.com/JNOSQL/diana/blob/master/diana-api/src/main/java/org/jnosql/diana/api/CloseResource.java
Sorry for my mistake.


chore: deprecate ipfs fns

Summary
No functional changes, just marking IPFS related fns as deprecated and removing mention to it from README.

recheck

Post merge review please.


Remove v1beta1 diff compare

Thanks for this work! I am also trying to upgrade v1 CRD in a more formal way, will also leverage your work here! Thanks

Pls help to check https://github.com/microsoft/frameworkcontroller/pull/69 and https://github.com/microsoft/frameworkcontroller/pull/72, meanwhile let's close this duplicated one.


Creating a new tenant runs the tenant's migrations on the system database connection

Hi,
I have recently upgraded from 5.3 to 5.4 I don't know if it may be related but I'm facing this strange problem that is driving me crazy.. basically i run migrate:fresh and then db:seed where in the DatabaseSeeder I am creating a Tenant, using the standard procedure described here: https://tenancy.dev/docs/hyn/5.4/creating-tenants
What happens is that the tenant's database is created, the migrations table is created correctly inside it, but the actual migrations are run on the system database.
I have checked what happens under the hood and in fact the database.default is set to the tenant connection which is correctly set to the current tenant.
I am using a pretty standard configuration, with only a custom Website model that extends directly the package's Website model.
I really don't know where to dig to debug this, maybe there is some other package interfering with multi-tenant and there is a listener resetting the connection to 'system'?
Any help is greatly appreciated, if you need more info i will post the files needed.

I was just about posting the same issue, waiting for any help.

Did anything change in terms of how laravel sets the connection on those commands? I'd have to dive into this to know for sure.

hmm.. I've been using 5.4 for quite some time now in my CI/CD setup, but all seems to be working just fine. I'm not sure if it's related but are you using a command to create the tenant?

After spending a day debugging, my issue was because of calling Schema::defaultStringLength(191) from the boot method of AppServiceProcider, when I remove it it worked just great, honestly, I don't know why this method caused the issue, maybe it initializes the connection to the system somewhere.

Thanks! Anyway I have rebuilt the app with the latest versions of Laravel and multi-tenant and everything is working as expected :) so i'm closing the issue!


Write a program  to make such a pattern like right angle triangle with a number which will repeat a number in a row.

Expected Output ;
1
22
333
4444

can my pull request is accepted but you didn't put hacktoberaccepted label on it @rahulkumar6142998


Listener: fix a crash when updating any listener that does not bind to port

Commit Message:
Fix the crash caused by listener update when the any listener set bind_to_port to false.
A listener does not bind to port has no io handle. close() already check it.
Now add the check to isOpen().
Also the Tcp(or Udp)ListenSocket::duplicate() creates parent ListenSocketImpl
Make it returning the derived ListenSocket, either TcpListenSocket or UdpListenSocket
Additional Description:
Risk Level:
Testing: Unit test on duplicate. Integration test on bind_to_port.
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

@lizan since this is blocking istio-proxy from picking up the change.

ref: https://github.com/istio/istio/pull/34188

/retest

@mathetake is OOO
Yes we should backport this to 1.20.x


Fix: Description hint field shows previous data after reenabling 'Req…

Details
Fixed Issues
$ https://github.com/Expensify/App/issues/52399
PROPOSAL: https://github.com/Expensify/App/issues/52399#issuecomment-2485009204
Tests
Same QA step

[x] Verify that no errors appear in the JS console

Offline tests
QA Steps

Open the Expensify app.
Navigate to Workspace Settings > Categories.
Click on any category.
Enable "Require description".
Click on Description hint.
Enter any value and save it.
Disable "Require description".
Re-enable "Require description".
Note that the Description hint field is empty.
Dismiss the RHP (Right Hand Panel).
Click on the same category again.
Verify that the Description hint field remains empty.

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
https://github.com/user-attachments/assets/fc1959a1-5910-48d6-929d-ed00e8023e04

Android: mWeb Chrome
https://github.com/user-attachments/assets/05bec6b4-97c7-4950-9623-65170ca7092c

iOS: Native
https://github.com/user-attachments/assets/92587703-1d4d-4f51-91b9-27401e256553

iOS: mWeb Safari
https://github.com/user-attachments/assets/d32d9416-bdee-475f-8035-306d5d0e01e0

MacOS: Chrome / Safari
https://github.com/user-attachments/assets/0e963c51-7cc2-4a22-9ef2-9e9bb2538456

MacOS: Desktop
https://github.com/user-attachments/assets/70f3afe3-41ee-4e7a-9b40-6779ac6753a6

Checking now...

@parasharrajat Thank you. Let me know if you have any questions?

@parasharrajat it's done

@parasharrajat I’ve already merged the main branch. Thanks!

@parasharrajat Do you have any updates? If you’re still facing issues with the Android build, please reach out on Slack to find a solution

@lakchote Could you please review this pull request? Thank you.


A pull of data from each node will populate the CDA release (snapshot on a given release)

Because we are not doing versioning in this release, a pull from GDC, PDC and IDC will occur on a given date prior to the production release.  Those dates will be made public in our documentation,

Donavan did this days before the release.

closing see data release notes https://cda.readthedocs.io/en/latest/ReleaseNotes.html#datasets-fields


Find DBT queries

In information schema jobs, identify the ones with the dbt comment header thing
and maybe parse it as Json? I need to make sure that if the dbt header fields change, it won’t goof up the output field schema as struct
maybe set up a macro to publish a UDF to parse the dbt stuff?

here's an example header for reference
(driven by this: https://github.com/bbrewington/gatech-cse6242-citibike/actions)
/* {"app": "dbt", "dbt_version": "1.3.0", "profile_name": "citibike_dbt", "target_name": "prod", "connection_name": "cse-6242-sp22-nyatl.information_schema"} */

    with tables as (
        select
            sql stuff blah blah blah

Here's a reproducible example:
with j as (
  select parse_json('{"app": "dbt", "dbt_version": "1.3.0", "profile_name": "citibike_dbt", "target_name": "prod", "connection_name": "cse-6242-sp22-nyatl.information_schema"}') as dbt_header
)

select
  string(dbt_header.app) as app,
  string(dbt_header.dbt_version) as dbt_version,
  string(dbt_header.profile_name) as profile_name,
  string(dbt_header.target_name) as target_name,
  string(dbt_header.connection_name) as connection_name
from j

with a as (
  select concat(
    '{"app": "dbt", "dbt_version": "1.3.0", ',
    '"profile_name": "citibike_dbt", "target_name": "prod", ',
    '"connection_name": "cse-6242-sp22-nyatl.information_schema"}\n\n\n\n',
    'select col1, col2 from table1 left join table2 on table2.a = table1.a'
  ) as object
),
split_vals as (
  SELECT struct(
    trim(split(b, ': ')[offset(0)], '"') as key,
    trim(split(b, ': ')[offset(1)], '"') as value
  ) as dbt_header
  from a
  cross join unnest(REGEXP_EXTRACT_ALL(object, r'"[\w\d\.\-_]+": ?"[\w\d\.\-_]+"')) as b
)
select *
from split_vals


RFC: Replacement of Poor Software Engineering courses with NPTEL

Problem:
Object Oriented Design, Design Patterns, Software Architecture teach their topics in roundabout ways by forcing students to use the android development toolkit. As such, they are not great for the purposes of the OSSU.
Software Engineering - Introduction is generally thought of as students here to be a poor course, which some people skipping it entirely.
(See this, #747, this, this, this)
Duration:
1 Month.
Background:
Splitting #1080
There are certain courses in the curriculum that aren't particularly great but have been kept due to lack of better courses. I've been scouring the internet and I found NPTEL (National Programme on Technology Enhanced Learning) which contains a lot of computer science courses from The Indian Institute of Technology - the highest tech universities in India. Think of it like OCW for the IITs.
They are generally though to be high quality courses in India, and this is the best university here. The downside is that there are not many practical assignments on NPTEL unlike their university counterparts.
Proposal:
Replace Object Oriented Design, Design Patterns, Software Architecture, and Software Engineering: Introduction with Software Engineering by IIT Kharagpur and Software Conceptual Design by IIT Bombay.
Alternatives:

We could look for alternative resources.

Alternative (I think some of you might prefer these to the NPTEL videos):
Object Oriented Design: https://www.youtube.com/watch?v=fJW65Wo7IHI&list=PLGLfVvz_LVvS5P7khyR4xDp7T9lCk9PgE
Design Patterns: https://www.youtube.com/watch?v=v9ejT8FO-7I&list=PLrhzvIcii6GNjpARdnO4ueTUAVR9eMBpc
Software Architecture: https://www.youtube.com/watch?v=3bxAm3XIFmk&list=PLdsOZAx8I5umhnn5LLTNJbFgwA3xbycar&index=146 (Should be elective)
Agile: https://www.youtube.com/watch?v=dfR6UBEBVvo&list=PLWKjhJtqVAbm0gPNunWf4pr4xV8I3Geh-
This are highly regarded youtube series.

What do you think of these resources
https://openlearninglibrary.mit.edu/courses/course-v1:MITx+6.005.1x+3T2016/about
https://openlearninglibrary.mit.edu/courses/course-v1:MITx+6.005.2x+1T2017/about

Just a prima facie review:
They overlap with HtC and some other courses. They don't cover design
patterns and software engineering. As such they are not suitable
replacement for the ones I proposed.
On Fri, 14 Oct, 2022, 22:41 Aayush SInha, @.***> wrote:

What do you think of these resources
https://openlearninglibrary.mit.edu/courses/course-v1:MITx+6.005.1x+3T2016/about
https://openlearninglibrary.mit.edu/courses/course-v1:MITx+6.005.2x+1T2017/about
—
Reply to this email directly, view it on GitHub
https://github.com/ossu/computer-science/issues/1082#issuecomment-1279263718,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL6AMA2D7IRGO6X7VLI4MLTWDGH3TANCNFSM6AAAAAAQ7OYVVY
.
You are receiving this because you authored the thread.Message ID:
@.***>

my prime recommendations will be above two MIT courses with this newer Software engineering course by IIT KGP
https://nptel.ac.in/courses/106105182

I had a deeper look into the MIT courses. Here is a comparison.
The topics covered by the courses we are removing:

 Course
Topics

 Object-Oriented Design

 Object-Oriented Design
Object-Oriented Analysis And Design
Unified Modeling Language (UML)

 Design Patterns
 Design Patterns

 Software Architecture
 Software Archetecture (subroutines, classes, functions, etc), evaluation and tradeoffs

 Software Engineering - Introduction
Agile development, REST and Async programming, design, refactoring, information security

The topics covered by the MIT courses are:

Course
Topics Covered

  6.005.1x Software Construction in Java
Java programming Software testing Code specifications Abstract data types

  6.005.2x Advanced Software Construction in Java

(1) grammars, parsing, and recursive datatypes;
(2) concurrent programming with threads.

They don't seem to be covering the same topics so it would not be productive to use them as replacements of the courses proposed to be removed.

I would like to quote CS2013 requirements for software development fundamentals and software engineering.
CS2013 says following topics are tier 1 topics that needs to be covered
SDF/Algorithms and Design
SDF/Fundamental Programming Concepts
SDF/Fundamental Data Structures
SDF/Development Methods
while we already cover algorithms and data structures in core theory

SDF/Fundamental Programming Concepts
[10 Core-Tier1 hours]
This knowledge unit builds the foundation for core concepts in the Programming Languages Knowledge Area, most notably in the paradigm-specific units: Object-Oriented Programming, Functional Programming, and Event-Driven & Reactive Programming.
Topics:
• Basic syntax and semantics of a higher-level language
• Variables and primitive data types (e.g., numbers, characters, Booleans)
• Expressions and assignments
• Simple I/O including file I/O
• Conditional and iterative control structures
• Functions and parameter passing
• The concept of recursion

SDF/Development Methods
[10 Core-Tier1 hours]
This unit builds the foundation for core concepts in the Software Engineering knowledge area, most notably in the Software Processes, Software Design and Software Evolution knowledge units.
Topics:
• Program comprehension
• Program correctness
o Types of errors (syntax, logic, run-time)
o The concept of a specification
o Defensive programming (e.g. secure coding, exception handling)
o Code reviews
o Testing fundamentals and test-case generation
o The role and the use of contracts, including pre- and post-conditions o Unit testing
• Simple refactoring
• Modern programming environments
o Code search
o Programming using library components and their APIs
• Debugging strategies
• Documentation and program style

Using CS2013 guidelines we can say above two MIT courses will be best suited to cover these topics

Lets talk about software engineering
cs2013 states following topics as absolute necessary

SE/Software Processes
Topics:
[Core-Tier1]
• Systems level considerations, i.e., the interaction of software with its intended environment (cross- reference IAS/Secure Software Engineering)
• Introduction to software process models (e.g., waterfall, incremental, agile) o Activities within software lifecycles
• Programming in the large vs. individual programming

SE/Software Design
[3 Core-Tier1 hours; 5 Core-Tier2 hours]
Topics:
[Core-Tier1]
• System design principles: levels of abstraction (architectural design and detailed design), separation of concerns, information hiding, coupling and cohesion, re-use of standard structures
• Design Paradigms such as structured design (top-down functional decomposition), object-oriented analysis and design, event driven design, component-level design, data-structured centered, aspect oriented, function oriented, service oriented
• Structural and behavioral models of software designs
• Design patterns

SE/Requirements Engineering
[1 Core-Tier1 hour; 3 Core-Tier2 hours]
The purpose of requirements engineering is to develop a common understanding of the needs, priorities, and constraints relevant to a software system. Many software failures arise from an incomplete understanding of requirements for the software to be developed or inadequate management of those requirements.
Specifications of requirements range in formality from completely informal (e.g., spoken) to rigorously mathematical (e.g., written in a formal specification language such as Z or first-order logic). In practice, successful software engineering efforts use requirements specifications to reduce ambiguity and improve the consistency and completeness of the development team’s understanding of the vision of the intended software. Plan-driven approaches tend to produce formal documents with numbered requirements. Agile approaches tend to favor less formal specifications that include user stories, use cases, and test cases.
Topics:
[Core-Tier1]
• Describing functional requirements using, for example, use cases or users stories
• Properties of requirements including consistency, validity, completeness, and feasibility

Now for software engineering I am offering this course by IIT KGP NPTEL
https://nptel.ac.in/courses/106105182
What this course covers
Week 01 : Introduction
Week 02 : Life Cycle Models I
Week 03 : Life Cycle Models II
Week 04 : Requirements analysis and specification
Week 05 : Basics of software design
Week 06 : Procedural design methodology
Week 07 : Object-oriented concepts
Week 08 : Introduction to UML: Class and Interaction Diagrams Week 09 : Object-oriented analysis and design
Week 10 : Testing I
Week 11 : Testing II
Week 12 : Testing III
The above course already covers topics like Object oriented design,Unified Modeling Language (UML) but also Procedural design methodology with software testing and secure programming. I guess this is best course that we have now to cover software engineering.
Not that I am against courses that you recommend its just that we dont really need them. Maybe an advanced software engineering section or advanced programming can get those courses.

Also sorry to mention about RFC #933 was accepted without considering CS2013 document.

Also sorry to mention about RFC https://github.com/ossu/computer-science/issues/933 was accepted without considering CS2013 document.

???

Also sorry to mention about RFC #933 was accepted without considering CS2013 document.

???

The RFC never made a reference to CS2013 doc also I did not find any course in CS2013 which was similar to software architecture although object oriented design and design patters were justified for SE/software design (might be a misunderstanding from my side , I apologise for it). The RFC just compared courses from original software engineering courses.

Using CS2013 guidelines we can say above two MIT courses will be best suited to cover these topics

Ah it looks like I should have been clearer in the earlier messages.
To elaborate on:

They overlap with HtC and some other courses. They don't cover design patterns and software engineering. As such they are not suitable replacement for the ones I proposed

What I meant to say was that the MIT courses don't teach the topics that are taught by the courses we are removing, and the topics they do teach are already covered by other courses in the curriculum.
I'll mark the course (in bold) in the curriculum that already covers the topic in the list:
SDF/Fundamental Programming Concepts
[10 Core-Tier1 hours]
This knowledge unit builds the foundation for core concepts in the Programming Languages Knowledge Area, most notably in the paradigm-specific units: Object-Oriented Programming, Functional Programming, and Event-Driven & Reactive Programming.
Topics:
• Basic syntax and semantics of a higher-level language **PLABC**
• Variables and primitive data types (e.g., numbers, characters, Booleans)  **PLABC**
• Expressions and assignments  **PLABC**
• Simple I/O including file I/O **CS50/Nand2Tetris**
• Conditional and iterative control structures  **PLABC, Nand2Tetris, Intro to CS**
• Functions and parameter passing **PLABC**
• The concept of recursion **HtC, PLABC**

SDF/Development Methods
[10 Core-Tier1 hours]
This unit builds the foundation for core concepts in the Software Engineering knowledge area, most notably in the Software Processes, Software Design and Software Evolution knowledge units.
Topics:
• Program comprehension **PLABC**
• Program correctness **Software Testing and Debugging**
o Types of errors (syntax, logic, run-time) **Software Testing and Debugging**
o The concept of a specification **Software Testing and Debugging**
o Defensive programming (e.g. secure coding, exception handling) **Core Security**
o Code reviews **Software Testing and Debugging**
o Testing fundamentals and test-case generation **HtC, Software Testing and Debugging**
o The role and the use of contracts, including pre- and post-conditions o Unit testing **Software Testing and Debugging**
• Simple refactoring **Software Testing and Debugging** (if I remember correctly)
• Modern programming environments **The missed Semester in CS** (???)
o Code search **Software Engineering** (???)
o Programming using library components and their APIs **HtC, Nand2Tetris**
• Debugging strategies**Software Debugging**
• Documentation and program style **HtC, Software Testing and Debugging**

As a result, by adding these two MIT courses, we are just making the curriculum redundant and longer but not covering any new topics.

Now for software engineering I am offering this course by IIT KGP NPTEL https://nptel.ac.in/courses/106105182

I agree with this change, it covers the same topics as the one I proposed as well. We still need to cover UML so we need to include the other one I proposed too (it's much shorter).

The one you mentioned @aayushsinha0706 seems to only cover the topics covered by Software Engineering - Introduction.
The other one also covers design patterns, object oriented design, and architecture along with software engineering.
The only thing is it doesn't cover UML, so I also included Software Conceptual Design to cover that part of the curriculum. Although my friends in the industry tell me "no one uses UML anymore" so maybe we can do without it?

I support the other one because it covers SE/Software Project Management which I didn't mentioned but it is a core-tier 2 topic that one I mentioned doesn't teach.
I guess final offering will be then:
Software Engineering IITB NPTEL
Software Testing and Debugging
in
core-application

Although above software engineering course is a video only course and I would request to add some supplementary reading material with it.
https://software-engineering-book.com
This is one of highly regarded textbooks as first course in software engineering (here is the contents  page of textbook), although author has not made textbook free but he has made slideshare presentations for each chapter of the book available for free which can be a good reading resource
https://software-engineering-book.com/slides/ and also listed some videos he finds useful to understand topics in book better.

End of 30 days.
Conclusion: We did not receive enough comments from students and no consensus seems to have been reached, so I've closed the RFC with no commits.


Distinguish sources from references in the codebase

Currently we define sources and references using the same class:
class Source:
    reference: tuple[str] = ()
    apply_transform_regex: str | None = None
    apply_transform_separator: str | None = None

Source.reference is a tuple indicating:

Another field: whose UID would be the concatenation of all the str in reference.
A CSV/JSON column: the UID is reference[0] and the column name is reference[1].

Problems:

Naming: sources and references are here used with very different meanings.
Source.reference is ambiguous and hard to use.

Instead, we would like something like:
class Origin:
    origin_uid: str  # UID in the structure graph
    column: str | None = None  # column if the origin references a CSV or a JSON.
    apply_transform_regex: str | None = None
    apply_transform_separator: str | None = None

Using something like Origin sounds good to me!

Another proposal is to remove reference and have only source in the JSON-LD format. So source could be a list. The first element is always the left, and all other elements are joined.


Abbreviation Detector chunker

The abbreviation detector currently breaks the incoming judgment into five chunks, which is sufficient for the vast majority of judgments. For longer judgments (or for very long ones, like ACL Netherlands) the number of chunks needs to be increased to prevent a memory error.

@imanehaf yes agree, should update it to reflect the size of the judgment


Event Request Approved almost 1 year ago, but not yet deliverd

Hi,
Could you please check the event request https://github.com/microsoft/ALAppExtensions/issues/19719 .
It was approved 1 year ago and it is not delivered.
Thank you!

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 11478.


[ship github 4] squash merge with Github API

This just adds the needed code to the code hosting driver. Wanted to break this PR in two to make it easier to review.
Still todo is using this function if the user has a GITHUB_TOKEN environment variable. I was thinking of using the following flow in that case:

use the same steps as if squash merging locally (stopping before we push up the changes)
parse the commit to get the title and message
reset the main branch to its previous commit (removing the squash merge)
merge via the github api
pull changes on the main branch
continue with the rest of the steps (we can now safely delete the branch)

EDIT: the todos have been taken care of in #1006
#750

@kevgo bump on this PR as it received a number of updates. Depending on our decision on #976, the structure of these updates to the drivers may change

looks good! Do you want to merge this or #976 first? I'm fine either way. Might be easier to merge this one first because its so much longer.

@kevgo this has been updated


update Workspace port to 2222

i was having problems with the port 22, is already in use on my PC.

@dilone Thanks for your contribution.
Do you have a reasoning behind using the port 2222 instead?
If it's only because of conflict, I don't think we should use the alternate ssh port by default, 22 is standard and if we follow the config for our other images we should be using standard ports for forwarding.
That being said, if you've had an issue with conflicting ports maybe we can address it in the debugging section of the documentation?

i have ssh server running on my Fedora, taking the port 22 by default, changing the map to the 2222 works great for me, all the command runs the same way, i just think is good addition to the project.

@Mahmoudz what do you think? 22 is development centric since if you ssh into a server the 22 port is probably already taken but I have seen configs using 2222...

@philtrep @dilone it makes sense, seems changing the default 22 would solve it but I'm not sure if it will cause troubles to the current users!! I'll leave it for you guys to decide.

Okay I'll merge this in since it does make sense that you might already have the port 22 used by ssh on your host machine, we'll revise if we get too many issues about this.
Thanks again for your contribution @dilone :)

should we mention that somewhere in the docs!?

@Mahmoudz good point! I haven't seen any mention of actual ssh (only exec) in the doc but we should indeed add a mention.

@philtrep thanksss