Can I convert a siglip only and not a siglip based LLM?

Based on supported models, conversion of SigLIP to TRT is already done, but can I use it standalone for a SigLIP model only?

Yea you could instantiate the CLIP class directly with name or path of SigLIP model, I think you will need to make your own script for calling it though.

From: Billy Cao @.>
Sent: Thursday, June 6, 2024 12:28:29 PM
To: dusty-nv/NanoLLM @.>
Cc: Subscribed @.***>
Subject: [dusty-nv/NanoLLM] Can I convert a siglip only and not a siglip based LLM? (Issue #16)
Based on supported models, conversion of SigLIP to TRT is already done, but can I use it standalone for a SigLIP model only?
—
Reply to this email directly, view it on GitHubhttps://github.com/dusty-nv/NanoLLM/issues/16, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADVEGKY6TL5J5RK2YETRBJLZGCE23AVCNFSM6AAAAABI5CNRNGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMZTQNRYGI4DCMY.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

Do i have to convert siglip to TRT myself, or can nanollm handle the conversion if I supply a pytorch model from HF?

It should handle it internally for you, and will save the trt engine

From: Billy Cao @.>
Sent: Thursday, June 6, 2024 2:01:47 PM
To: dusty-nv/NanoLLM @.>
Cc: Dustin Franklin @.>; Comment @.>
Subject: Re: [dusty-nv/NanoLLM] Can I convert a siglip only and not a siglip based LLM? (Issue #16)
Do i have to convert siglip to TRT myself, or can nanollm handle the conversion if I supply a pytorch model from HF?
—
Reply to this email directly, view it on GitHubhttps://github.com/dusty-nv/NanoLLM/issues/16#issuecomment-2153104983, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADVEGK2JSQTU5HOY2GOMED3ZGCPYXAVCNFSM6AAAAABI5CNRNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTGEYDIOJYGM.
You are receiving this because you commented.Message ID: @.***>

Sorry another question, where is the code that does the inference on TRT engine? I am trying to write my own inference script but faced some issues. I thought you are experienced with TRT inference so i want to try my luck and see how you implemented it.
This is the script I have for context
class TRTInference:
    def __init__(self, engine_path, output_names_mapping: dict = None, fp16=True, verbose=False):
        self.engine_path = engine_path
        self.output_names_mapping = output_names_mapping or {}
        self.logger = trt.Logger(trt.Logger.VERBOSE) if verbose else trt.Logger(trt.Logger.INFO)
        self.engine = None
        self.load_engine()
        assert self.engine is not None, 'Failed to load TensorRT engine.'

        self.context = self.engine.create_execution_context()

        self.input_names = self.get_input_names()
        self.output_names = self.get_output_names()

        self.dtype = np.float16 if fp16 else np.float32

    def load_engine(self):
        with open(self.engine_path, 'rb') as f, trt.Runtime(self.logger) as runtime:
            self.engine = runtime.deserialize_cuda_engine(f.read())

    def get_input_names(self):
        names = []
        for _, name in enumerate(self.engine):
            if self.engine.get_tensor_mode(name) == trt.TensorIOMode.INPUT:
                names.append(name)
        return names

    def get_output_names(self):
        names = []
        for _, name in enumerate(self.engine):
            if self.engine.get_tensor_mode(name) == trt.TensorIOMode.OUTPUT:
                names.append(name)
        return names

    def get_bindings(self):
        Binding = namedtuple('Binding', ('name', 'dtype', 'shape', 'data', 'ptr'))
        bindings = OrderedDict()

        for i, name in enumerate(self.engine):
            shape = self.engine.get_tensor_shape(name)
            shape = tuple(shape)
            if any(s < 0 for s in shape):  # set dynamic axis to be 1
                shape = tuple(1 if s < 0 else s for s in shape)
            dtype = trt.nptype(self.engine.get_tensor_dtype(name))
            if self.engine.get_tensor_mode(name) == trt.TensorIOMode.INPUT:
                data = np.random.randn(*shape).astype(dtype)
                ptr = cuda.mem_alloc(data.nbytes)
                bindings[name] = Binding(name, dtype, shape, data, ptr)
            else:
                data = cuda.pagelocked_empty(trt.volume(shape), dtype)
                ptr = cuda.mem_alloc(data.nbytes)
                bindings[name] = Binding(name, dtype, shape, data, ptr)

        return bindings

    def __call__(self, blob):
        self.stream = cuda.Stream()
        input_bs = next(v for v in blob.values()).shape[0]
        blob = {n: np.ascontiguousarray(v) for n, v in blob.items()}
        for n in self.input_names:
            input_shape = self.engine.get_tensor_shape(n)
            input_shape = (input_bs, *input_shape[1:])
            self.context.set_input_shape(n, input_shape)
            d_input = cuda.mem_alloc(np.random.randn(*input_shape).astype(self.dtype).nbytes)
            self.context.set_tensor_address(n, int(d_input))
            cuda.memcpy_htod_async(d_input, blob[n], self.stream)

        output_shape_list = []
        for n in self.output_names:
            output_shape = self.engine.get_tensor_shape(n)
            output_shape = (input_bs, *output_shape[1:])
            d_output = int(cuda.mem_alloc(np.random.randn(*output_shape).astype(self.dtype).nbytes))
            self.context.set_tensor_address(n, d_output)
            output_shape_list.append(output_shape)

        assert self.context.all_binding_shapes_specified

        self.context.execute_async_v3(stream_handle=self.stream.handle)

        outputs = {}
        for n, output_shape in zip(self.output_names, output_shape_list):
            output = np.empty(output_shape, dtype=self.dtype)
            cuda.memcpy_dtoh_async(output, self.context.get_tensor_address(n), self.stream)
            outputs[self.output_names_mapping.get(n, n)] = output

        self.stream.synchronize()

        return outputs

@aliencaocao I am using torch2trt from @jaybdub which gives you a model object with the same interface as pytorch so you can just transparently replace your pytorch model with the TRT version.  Which is what I do here:  https://github.com/dusty-nv/NanoLLM/blob/b0be32710f0faf7983e63c98b433e6203c061b57/nano_llm/vision/clip.py#L176
If I wanted to make something that didn't depend on pytorch, then yea I would use the TRT API directly, or onnxruntime if I wanted to be able to fallback to cuDNN.

Thanks for pointing out. Do you know how can I load a .engine directly using torch2trt? I have exported it separately to use my own shape configs.

Actually i managed to port it over but im getting some
[06/08/2024-12:35:15] [TRT] [E] 1: [convBaseRunner.cpp::nvinfer1::rt::task::CaskConvBaseRunner::execute::300] Error Code 1: Cask (Cask convolution execution)
This is probably not related but have you seen this before?

@aliencaocao not specifically as related to this (being CLIP/SigLIP)

Thank you. I will try to convert using torch2trt myself and see how.
I tried to use nanoLLM but it is missing the text model of siglip which I also need, else I would have used it straight away.
Thank you for your help!

OK thanks, let me know if you find that TRT or torch2trt can build/run the SigLIP text encoder, that would be good for me to add too.

I got the exporting for vision model to work, but not the text part due to HF transformers expecting a dict as input but torch2trt expect the tensor straight away.
File C:\Program Files\Python39\lib\site-packages\transformers\models\siglip\modeling_siglip.py:774, in SiglipTextTransformer.forward(self, input_ids, attention_mask, position_ids, output_attentions, output_hidden_states, return_dict)
    771 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    773 if input_ids is None:
--> 774     raise ValueError("You have to specify input_ids")
    776 input_shape = input_ids.size()
    777 input_ids = input_ids.view(-1, input_shape[-1])

ValueError: You have to specify input_ids

will see if there's another way before I get into patching HF transformers

Got it to work after https://github.com/NVIDIA-AI-IOT/torch2trt/pull/932

Full conversion script:
import torch
from torch2trt import torch2trt
from transformers import SiglipModel

model = SiglipModel.from_pretrained('siglip ckpt path/HF id', torch_dtype=torch.float16).cuda().eval()

text_model = model.text_model
vision_model = model.vision_model

dummy = torch.ones(1, 3, 384, 384, dtype=torch.float16, device='cuda')
model_trt = torch2trt(vision_model, [dummy], fp16_mode=True, min_shapes=[(1, 3, 384, 384)], opt_shapes=[(4, 3, 384, 384)], max_shapes=[(10, 3, 384, 384)], use_onnx=True)  # change the shapes here - for me I know max bs is 10 but you should probably not limit
y = vision_model(dummy).pooler_output
y_trt = model_trt(dummy)['pooler_output']
torch.save(model_trt.state_dict(), 'vision_trt.pth')
print('Vision model exported. atol:', torch.max(torch.abs(y - y_trt)))

dummy = torch.ones(1, 64, dtype=torch.long, device='cuda')  # siglip tokenizer should always pad to 64
model_trt = torch2trt(text_model, [dummy], fp16_mode=True, min_shapes=[(1, 64)], opt_shapes=[(1, 64)], max_shapes=[(1, 64)], use_onnx=True)
y = text_model(dummy).pooler_output
y_trt = model_trt(dummy)['pooler_output']
torch.save(model_trt.state_dict(), 'text_trt.pth')
print('Text model exported. atol:', torch.max(torch.abs(y - y_trt)))

Then can just load via TRTModule.load_state_dict(torch.load(os.path.join(clip_path, 'vision_trt.pth')))
To get the logits:
            vision_input = self.clip_image_processor(images=boxes, return_tensors='pt').to(self.device)
            text_inputs = self.clip_tokenizer(im_captions_templated, return_tensors='pt', padding='max_length', truncation=True).to(self.device)  # processor wont work since it dont pad to max_length=64
            vision_input = vision_input['pixel_values'].type(torch.float16)
            image_feat = self.clip_vision_trt(vision_input)['pooler_output']
            text_feat = self.clip_text_trt(text_inputs['input_ids'])['pooler_output']
            image_feat /= image_feat.norm(p=2, dim=-1, keepdim=True)
            text_feat /= text_feat.norm(p=2, dim=-1, keepdim=True)
            scores = image_feat @ text_feat.T * self.clip_logit_scale_exp + self.clip_logit_bias
            scores = scores.squeeze(-1).tolist()  # sigmoid not needed as it dont change the ranking

Thanks @aliencaocao , that's great!   I'm going to unify the various CLIP/SigLIP implementations I have floating around between NanoLLM/NanoDB with support for the text encoder in TRT alongside the vision encoder 👍

@aliencaocao did you get the text encoder working in TRT with real token ID's?   The output delta is small when the input_id's are all 1, but when I actually tokenize a real string it doesn't work.  Which version of TensorRT are you using?

TRT 10.1. Yes all tokenid, i am using it in on over 20k samples already

One very important thing is do not use HF Processor but instead HF tokenzier. The processor does not pad input to 64 ('max-length') token which is what siglip has been trained on. also double check that padding token id is 1.
This differs from other CLIPs where the padding would be the max length in a batch and not always 64/whatever context length.
Attenmask is not needed.
And to get the same output as HF pipeline (python/js), you need to add a torch.sigmoid(scores) before the last line. For purpose of image reranking that's not needed since sigmoid wont change the order.


2.1.0 collector does not collect

I request 1000 times,but the web only list  five.what's happen. run it with the doc and only replace the jar,no error and no data ,the other requests had gone way? the port is still listen and hbase work well

its your agent...
change this value in pinpoint-env.config inside the profiles/release folder
from:
profiler.sampling.rate=20
to
profiler.sampling.rate=1
or call it directly from your run command
java -Dprofiler.sampling.rate=1 -jar balbalblaba.jar
or add it in the environment variables JAVA_OPTS or CATALINA_OPTS
JAVA_OPTS=$JAVA_OPTS -Dprofiler.sampling.rate=1

thanks


Update client3:core, client3:zio1 to 3.8.12

Updates

com.softwaremill.sttp.client3:core
com.softwaremill.sttp.client3:zio1

from 3.8.9 to 3.8.12.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.softwaremill.sttp.client3" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.softwaremill.sttp.client3" }
}]

labels: test-library-update, early-semver-patch, semver-spec-patch, version-scheme:early-semver, commit-count:1

Superseded by #304.


Enh dummy cl

In case "Enh dummy cl" isn't self-evident (ahem), this is about making ophyd usable without pyepics installed, particular for doing stuff with ophyd.sim in the context of analysis or prototyping.


Update all the things

This addon was throwing some deprecations and errors in other addons/apps using it, so I wanted to update everything and get it working on modern Ember.

@alexander-alvarez this updates and fixes a lot of things. If you could please approve the workflow, the tests should run.

There is one pesky test failure / issue I am encountering. Still trying to debug it, but this PR almost fully updates all the things 😃


Incorrect diff for the expect.contains assertion

What is your Test Scenario?
Use expect.contains to compare two string values.
What is the Current behavior?
An incorrect diff shown (pay attention to the error message and the diff):
AssertionError: expected 'test' to include 'foo'

      + expected - actual

      -'test'
      +undefined

What is the Expected behavior?
AssertionError: expected 'test' to include 'foo'

      + expected - actual

      -'foo'
      +'test'

What is your web application and your TestCafe test code?

Your website URL (or attach your complete example):

Your complete test code (or attach your test files):

 fixture`1`.page`./index.html`

test(`1`, async t => {
    await t.expect('test').contains('foo');
});

Your complete configuration file (if any):

Your complete test report:

Screenshots:

Steps to Reproduce:

Run the test
Your Environment details:

testcafe version:                   v1.9.1
node.js version:                    v14.1.0
command-line arguments:  testcafe chrome test.js
browser name and version: Chrome 84
platform and version:          Windows 10
other:

t.expect.notOk method result shown in diff as undefined instead of false:
AssertionError: expected true to be falsy

      + expected - actual

      -true
      +undefined

      Browser: Chrome 84.0.4147.135 / Windows 10

         1 |fixture `Getting Started`
         2 |    .page `http://devexpress.github.io/testcafe/example`;
         3 |
         4 |test('Should be failed', async t => {
       > 5 |    await t.expect(true).notOk();
         6 |});
         7 |


7ameed patch 1

@shieldfy details

@shieldfy details Secret detected in /core/passport.js in line 84

@shieldfy details Secret detected in /core/passport.js in line 84

@shieldfy fix Secret detected in /core/passport.js in line 84

@shieldfy fix Secret detected in /core/passport.js in line 84

@shieldfy fix Secret detected in /core/passport.js in line 84


Add History generator

A generator that stores a size-limited history of encountered values.

Done in #3f2784d.
Now add the documentation.


Give the user system administrator privileges, it also automatically checked the project manager permissions

In GitLab by @sokril on Nov 8, 2017, 09:15

In GitLab by @yhua123 on Nov 8, 2019, 07:40
by design

In GitLab by @sokril on Nov 8, 2019, 08:13
closed


Generated archetype docs should link to Python and Rust API docs

We cannot link to the python docs, because there is no way to link to the latest python docs

Blocked by https://github.com/rerun-io/rerun/issues/3647


Dotted arrow from OTel Collector Agent to Host is reversed

The reference architecture diagram on the landing page of OpenTelemetry documentation has some arrows, and a couple of them should be reversed.
The orange dotted arrow from OTel Collector Agent to Host should be the one from Host to OTel Collector Agent.

If these arrows are correct and as intended, still the arrows are confusing and I think we need an explanation of what "Host" is.

This is now out of date I think, since we don't use this figure anymore on the site.

If you want, I can ask Ella (the designer) to update the diagram, I like these. Please let me know.


Iac terraform collector boot

Infrastructure As Code : Change Release Details
What is this release all about : A new Widget to showcase the Terraform Cloud App details is designed. This code is written as a backend controller and collector.
https://app.terraform.io/session
Do we have also forked the UI and the core
UI : https://github.com/prash897/UI
Core : https://github.com/prash897/hygieia-core-1
The Classes and the description
Terraform Controller: To expose the api calls made from UI to get the Component Data
TerraformCollectorTask : As with the framework , implemeting Collector Task to regulary collect the details from Terraform Cloud App and feed into the MongoDb
TeraafromServiceImpl: To populate the componentData mdel and send back to UI
TerraformCustomRepository: A MongoTemplate based repository, to do some aggrrgate group by queries against mondo db
Each Terraform app has Organization > Workspace > run jobs
Organization : Model to store the Organization details from Terraform Cloud App
Workspace : Model to Store Workpsace belonging to an Organizatiion from Terrafrom Cloud App
Run : Model to Store Runs belonging to an Workpsace from Terrafrom Cloud App
TerraformCollectorItem: The class extending the CollectorItem storing only the apiToken, which is enough to call the Terraform Cloud Apis

Affects: <jira-feature-collector-version-number>.

Individual Agreement
Corporate Agreement

[ ] I Have signed the CLA

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: Prashant Chouhan - (Service Transformation):x: prash897Prashant Chouhan - (Service Transformation) seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

@prash897 can you please create a new PR as this PR is corrupted and has multiple committers.
 Please let me know if you have more questions in this regard.


Compilation error: Declaration of 'AFHTTPClient' must be imported from module 'Chute_SDK.GCClient' before it is requied.

Hi, first of all I tested sample app and it works. Then I've added it to my Podfile, which looks like that:
use_frameworks!

pod 'PhotoPickerPlus'

post_install do |installer|
  installer.pods_project.build_configuration_list.build_configurations.each do |configuration|
    configuration.build_settings['CLANG_ALLOW_NON_MODULAR_INCLUDES_IN_FRAMEWORK_MODULES'] = 'YES'
  end
end

Then I ofc did pod install and everything succeded. After opening xcode I also cleaned & builded it.
Also important note: my project uses swift language and latest ios sdk (8.4).
Here's an error:
Declaration of 'AFHTTPClient' must be imported from module 'Chute_SDK.GCClient' before it is requied.

Also note that it's error inside framework I can't even edit this file at all.
I'm stuck. How to fix that?

Ok I figured out that errors goes away when I remove "use_frameworks!" line from Podfile. However by removing this I'm no longer able to use any objc module from cocoapods (it's making objc-swift bindings).

thanks for reaching out, we will need to investigate this and will let you know, i know we've done similar stuff with it in a project.

Thanks for response. I've managed to create bridging header by myself, which was actually very simple:
#ifndef Photos_PhotoPickerPlus_Bridging_Header_h
#define Photos_PhotoPickerPlus_Bridging_Header_h

#import "PhotoPickerViewController.h"

#endif

App builds without errors however it's not working properly:
Here's orginal sample project:

Now here's the same controller (tested on ipod touch 5g and iphone 6 simulator):

        var picker = PhotoPickerViewController()
        picker.isMultipleSelectionEnabled = true

        self.presentViewController(picker, animated: true, completion: nil)

There are missing labels for sections, missing icons for take photo and all media and what's the worst these 2 first buttons doesn't do anything. Just tapping them do nothing.

Damn it's not fault of swift I think. I've just created empty objc project added the same code (translated into objc) what I had in swift and it works exactly the same.
But I've added configuration file with proper app_id and app_secret (the same pair worked in sample app).


Bug in FileUtils while creating URI on Windows

Hi !
There is a bug in the method :
    // org.pf4j.util.FileUtils line 216
    public static Path getPath(Path path, String first, String... more) throws IOException {
        URI uri = path.toUri();
        if (isJarFile(path)) {
            uri = URI.create("jar:file:" + path.toString());
        }

        return getPath(uri, first, more);
    }

You must force "\" to "/" while doing path.toString() for Jar files.
The error is :
Illegal character in opaque part at index 11: jar:file:D:\Dev\WorkingDir\projectName\plugins\plugin-pluginName.jar

Use 2.1.0-SNAPSHOT.

Thanks for your reactivity :)


"More Like This" feature

Do you have any plans to implement a "More Like This"[1, 2] feature?
[1] https://lucene.apache.org/core/8_4_1/queries/org/apache/lucene/queries/mlt/MoreLikeThis.html
[2] https://whoosh.readthedocs.io/en/latest/api/searching.html?highlight=more like#whoosh.searching.Hit.more_like_this

Hey, this is not currently on the roadmap. MeiliSearch do have pagination if you want to have more results based on a single search but that's it.
I think it may add complexity when designing a search-as-you-type box. The search results should be found within two or three words in a query, thus reachable with pagination.
Can you provide a use-case?

Hello @tpayet,
My use case is pretty simple. I am using a "More Like This" to implement a very crude item-to-item [1] product recommendation based on a few actions a user can perform.
e.g:
"Product A" is similar to "Product B", if the user "Likes" "Product A", she might also like "Product B."
Currently, I am using Whoosh [2] to perform this task; however, I am also planning to integrate Meilisearch in this project, it would be very helpful to have such a feature on Meilisearch so I could remove my dependency on Whoosh.
I guess I can implement something similar with Meilisearch by somehow extracting the "Key terms" of a document and performing a search with the terms.
[1] https://en.wikipedia.org/wiki/Item-item_collaborative_filtering
[2] https://whoosh.readthedocs.io/en/latest

I understand ^^
It is not currently on our roadmap, but if you are willing to implement something, @Kerollmops can help you and guides you through the understanding of MeiliSearch :)
Let me know if you are interested, we can invite you on a Slack, it will be easier for us to collaborate

@tpayet I could try, but, besides a few experiments, I really don't have a good experience with Rust.

Hey @dimiro1,
I though a little bit about this "More Like This" feature.
As I understand it, to find documents that looks like a given document or a group of documents we could take all the rarest words in the origin document or group of documents and search for all other documents that have those words too (remove the most common words when not enough documents are found).
The problem with this approach is that there is not way to get all the words related to a given document, MeiliSearch use an inverted index: the key is the word and the value is the list of documents ids containing the given word.
So retrieving the words contained in a document or a group of documents can only be done by iterating through all the words and searching for those that are in the documents. It can take an huge amount of time as it is O(n * m) where n is the number of words and m is the size of all the list of documents ids associated with each words.

@Kerollmops Thanks for your reply, your explanation makes sense to me. A possible solution would be to start storing the keywords together with the documents. Unfortunately, this has the disadvantages of increasing the index size and will require some sort of transaction mechanism to make sure the keywords are in sync.

Hello, Thanks for this feature proposal!
I will close this issue in favor of our public roadmap.
I invite everyone interested in this feature to update it on the roadmap.


MacInterface and IpInterface Instantiation

Existing definitions for Equipment and Instantiations have to be completed by definitions for the instantiation of MacInterfaces and IpInterfaces.
Potential Solution:
MacInterfaces and IpInterfaces get automatically instantiated in corresponding numbers as soon as ExpectedEquipment, which offers such interfaces, gets instantiated.

regarding IP - it is deferred for future use - 2.1

for IP a new issue should be created to be addressed in future.
Please see https://github.com/openBackhaul/equipment/issues/27

regrading MacInterface

please see MAC related paragraphs of Equipment #7

The following sentence should be considered to address this issue:
"As soon as an ExpectedEquipment object, which is representing an Ethernet Switch with four internal interfaces, has been instantiated, the device shall automatically instantiate four LTP(EthernetContainer), four LTP(MacInterface) and four LTP(VlanInterface) (only in case the device supports VLAN bridging)."

Proposal made to the 5G-xhaul call on 14th of December 2022
The issue should be spitted into 2 parts - one for IPInterfaces and one for MacInterfaces
The IpInterface part should be covered by https://github.com/openBackhaul/equipment/issues/27
The MacInterface part was already covered by https://github.com/openBackhaul/equipment/issues/7
This issue should be closed without further activity.


Improve error message for mesheryctl mesh

Signed-off-by: PratikforCoding<EMAIL_ADDRESS>Notes for Reviewers
This PR fixes #6270
Signed commits

[x] Yes, I signed my commits.

Codecov Report
Base: 16.78% // Head: 12.54% // Decreases project coverage by -4.24% :warning:

Coverage data is based on head (be1991b) compared to base (416427e).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6625      +/-   ##
==========================================
- Coverage   16.78%   12.54%   -4.25%
==========================================
  Files         115      115
  Lines       11218    11218
==========================================
- Hits         1883     1407     -476
- Misses       8860     9497     +637
+ Partials      475      314     -161

Flag
Coverage Δ

gointegrationtests
?

unittests
12.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

mesheryctl/internal/cli/root/system/start.go
1.70% <0.00%> (-57.03%)
:arrow_down:

mesheryctl/internal/cli/root/system/status.go
1.12% <0.00%> (-53.94%)
:arrow_down:

mesheryctl/internal/cli/root/system/update.go
1.17% <0.00%> (-47.06%)
:arrow_down:

mesheryctl/internal/cli/root/system/restart.go
1.88% <0.00%> (-45.29%)
:arrow_down:

mesheryctl/internal/cli/root/system/stop.go
2.22% <0.00%> (-41.49%)
:arrow_down:

mesheryctl/internal/cli/root/root.go
24.24% <0.00%> (-40.91%)
:arrow_down:

mesheryctl/internal/cli/root/system/logs.go
3.26% <0.00%> (-36.96%)
:arrow_down:

mesheryctl/internal/cli/root/version.go
0.00% <0.00%> (-32.95%)
:arrow_down:

mesheryctl/internal/cli/root/system/check.go
1.03% <0.00%> (-23.03%)
:arrow_down:

mesheryctl/internal/cli/root/error.go
0.00% <0.00%> (-20.00%)
:arrow_down:

... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@nishantwrp review this please.

This was covered by #6626 along with some other commands.

Adding " (partial) " to PR description.

Thanks, @PratikforCoding !

Thanks @leecalcote sir!


[Remote Store] Cluster State Applier thread blocked on remote store operations

Describe the bug
On remote store clusters, we can see cluster state applier thread is blocked on remote store calls. In case when the calls to remote store takes a lot of time, the node is not able to apply the cluster state and LagDetector on the cluster manager kicks it out .
[2024-01-24T20:56:24,412][WARN ][o.o.i.c.IndicesClusterStateService] [a] [.index][5] marking and sending shard failed due to [failed to create shard]
java.io.IOException: java.io.IOException: Exception when listing blobs by prefix [x/y/z/metadata]
    at org.opensearch.index.store.RemoteDirectory.listFilesByPrefixInLexicographicOrder(RemoteDirectory.java:138)
    at org.opensearch.index.store.RemoteSegmentStoreDirectory.readLatestMetadataFile(RemoteSegmentStoreDirectory.java:191)
    at org.opensearch.index.store.RemoteSegmentStoreDirectory.init(RemoteSegmentStoreDirectory.java:145)
    at org.opensearch.index.store.RemoteSegmentStoreDirectory.<init>(RemoteSegmentStoreDirectory.java:132)
    at org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory.newDirectory(RemoteSegmentStoreDirectoryFactory.java:74)
    at org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory.newDirectory(RemoteSegmentStoreDirectoryFactory.java:49)
    at org.opensearch.index.IndexService.createShard(IndexService.java:488)
    at org.opensearch.indices.IndicesService.createShard(IndicesService.java:1036)
    at org.opensearch.indices.IndicesService.createShard(IndicesService.java:212)
    at org.opensearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:673)
    at org.opensearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:650)
    at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:295)
    at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:606)
    at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:593)
    at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:561)
    at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484)
    at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186)
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:858)
    at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
    at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Exception when listing blobs by prefix [x/y/z/metadata]
    at org.opensearch.repositories.s3.S3BlobContainer.listBlobsByPrefixInSortedOrder(S3BlobContainer.java:455)
    at org.opensearch.common.blobstore.BlobContainer.listBlobsByPrefixInSortedOrder(BlobContainer.java:234)
    at org.opensearch.common.blobstore.EncryptedBlobContainer.listBlobsByPrefixInSortedOrder(EncryptedBlobContainer.java:207)
    at org.opensearch.index.store.RemoteDirectory.listFilesByPrefixInLexicographicOrder(RemoteDirectory.java:127)
    ... 22 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
    at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
    at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)

On cluster manager node :
[2024-01-24T20:56:24,339][WARN ][o.o.c.c.LagDetector      ] [0f2] node [{a}{b}{c}{{dir}] is lagging at cluster state version [29651], although publication of cluster state version [29652] completed [1.5m] ago

[2024-01-24T20:56:25,192][INFO ][o.o.c.s.MasterService    ] [0f2] node-left [{a}{b}{c}{{dir}] reason: lagging], term: 14, version: 29656, delta: removed {[{a}{b}{c}{{dir}]}

Related component
Storage:Durability
To Reproduce
We see this when there are high amount of relocations in the cluster .
Expected behavior
For cluster state applier, we should have a dedicated threadpool, so that it doesn't get blocked on any resource - be it threadpool / connections etc.
Additional Details
Plugins
repository-s3
Host/Environment (please complete the following information):

Amazon Linux 2

Additional context
Add any other context about the problem here.

Thanks Gaurav, Ideally cluster state applier thread blocks or should block any blocking or networking operation on that thread. But given the flow with remote store we might need a dedicated and prioritized threadpool in remote store for cluster state applier interactions and we may want to still want to keep the blocking behaviour of the cluster state applier thread.

We shouldn't perform expensive operation in Cluster state applier thread. If you can offload this work to dedicated thread pool, that would be preferred.
Ideally applier are expected to finish before any listeners can be triggered but we can evaluate if appliers can execute their task which doesn't depend on the cluster state in the parallel in the background.


Product description doesn’t allow link on picture

Hi. When I use the Product Recommendation it allows to add a link on any text, a button, but it doesn’t allow to add a link to the picture. Is there any way I can do this? Thanks in advance.

Hi, as far as I know there is no way to add a link to the picture at the moment.
As a workaround you could create a Product card and then inspect your page and copy the generated HTML. You can paste that code in a HTML card and add a link sorounding the image manually.
I would also suggest to contact the Ghost support since this is not really related to the theme.

I will close this issue for now, please feel free to reopen it if needed.


rebuild for petsc 3.22

after this, there should be a solvable enviornment for latest complex petsc, mpc, and scifem on 0.9

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/recipe.yaml) and found it was in an excellent condition.


[BUG] Longhorn cifs backups cannot find credentials

Describe the bug
Longhorn won't find my credentials for cifs backups. Have also tried changing to nfs but it kept complaining about cifs credentials even though I had the correct credentials passed through.
To Reproduce
Hard to reproduce. Enter the cifs backup target in the gui also with the credentials.
Expected behavior
Should allow me to backup my volumes to my unraid server.
Support bundle for troubleshooting
supportbundle_913f7d1d-6412-427c-87ee-1fe64c171768_2024-05-24T02-39-44Z.zip
Environment
Longhorn version: 1.6.1
Impacted volume (PV): bazarr, deluge-config, deluge-data, flaresolverr, jfa-go, kavita, lidarr, overseerr, radarr, readarr, sonarr, vaultwarden, whisparr
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: K3s 1.27.10+k3s2
    Number of control plane nodes in the cluster: 3
    Number of worker nodes in the cluster: 3
Node config
    OS type and version: Debian Lunar
    Kernel version: 6.2.0-1018-kvm
    CPU per node: 4
    Memory per node: 6GB
    Disk type (e.g. SSD/NVMe/HDD): SSD
    Network bandwidth between the nodes (Gbps): 1Gbps
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Proxmox VMs
Number of Longhorn volumes in the cluster: 19

Additional context

Hi @ZanderPittari,
The backup target URL had been modified to nfs and credential secret had been removed.
I believe you have a SMB/CIFS server for backups, right?
The last error log in the longhorn-manager is
2024-05-24T12:38:41.273798359+10:00 E0524 02:38:41.270519 1545448 mount_linux.go:230] Mount failed: exit status 1
2024-05-24T12:38:41.273801804+10:00 Mounting command: mount
2024-05-24T12:38:41.273804717+10:00 Mounting arguments: -t cifs -o soft,<masked>,<masked> //<IP_ADDRESS>/longhorn-backups /var/lib/longhorn-backupstore-mounts/192_168_10_2/longhorn-backups
2024-05-24T12:38:41.273807308+10:00 Output: username specified with no parameter
2024-05-24T12:38:41.273809302+10:00

Could you provide the secret smbcreds or check if CIFS_USERNAME and CIFS_PASSWORD are set correctly? (ref: Set up SMB/CIFS Backupstore)
You could empty the Backup Target to clear the error message could not access s3 without credential secret first and reset the backup target URL and credential secret.
(a bit related to https://github.com/longhorn/longhorn/issues/8299)

I think I accidentally uploaded the wrong support bundle. Let me attach a new one:
supportbundle_913f7d1d-6412-427c-87ee-1fe64c171768_2024-05-24T04-05-43Z.zip

Still the same error, Could you provide the secret smbcreds or check if CIFS_USERNAME and CIFS_PASSWORD are set correctly? (ref: Set up SMB/CIFS Backupstore)
2024-05-24T14:06:11.397990263+10:00 time="2024-05-24T04:06:11Z" level=info msg="Mounting CIFS share cifs://<IP_ADDRESS>/longhorn-backups on mount point /var/lib/longhorn-backupstore-mounts/192_168_10_2/longhorn-backups with optio
ns [soft]" func="cifs.(*BackupStoreDriver).mount" file="cifs.go:115" pkg=cifs
2024-05-24T14:06:11.397994138+10:00 E0524 04:06:11.395709 1571963 mount_linux.go:230] Mount failed: exit status 1
2024-05-24T14:06:11.397997593+10:00 Mounting command: mount
2024-05-24T14:06:11.397999919+10:00 Mounting arguments: -t cifs -o soft,<masked>,<masked> //<IP_ADDRESS>/longhorn-backups /var/lib/longhorn-backupstore-mounts/192_168_10_2/longhorn-backups
2024-05-24T14:06:11.398002006+10:00 Output: username specified with no parameter
2024-05-24T14:06:11.398003433+10:00
2024-05-24T14:06:11.398005500+10:00 time="2024-05-24T04:06:11Z" level=fatal msg="Failed to run list system backup command" func=cmd.SystemBackupCmd.SystemBackupListCmd.func4 file="system_backup.go:72" error="cannot mount CIFS share <IP_ADDRESS>/longhorn-backups, options [soft]: mount failed: exit status 1\nMounting command: mount\nMounting arguments: -t cifs -o soft,<masked>,<masked> //<IP_ADDRESS>/longhorn-backups /var/lib/longhorn-backupstore-mounts/192_168_10_2/longhorn-backups\nOutput: username specified with no parameter\n"
2024-05-24T14:06:11.398007061+10:00 : exit status 1

Not sure it is about kernel issue of CIFS (https://github.com/longhorn/longhorn/issues/8611)

I know the creds work because I am using the same ones for other shares. The details are below:
CIFS_USERNAME= zander
CIFS_PASSWORD= 2iPr8cCWAP7rPz

The error occurred at list system backup for synchronizing the system backup but it would synchronize the backup volumes and backup backing images first (they should fail if the secret setting is incorrect.)
Could you upgrade to v1.6.2 to check if it will occur again because we have modified the flow to get the synchronizing information?

I just upgraded to 1.6.2 but now getting this error. Which looks similar to what you posted above:

Could you show the secret information by the command kubectl -n longhorn-system get secret smbcreds -oyaml?

This is what I get. The username and password are encrypted

Hi @ZanderPittari,
Please replace the data.username to data.CIFS_USERNAME and data.password to data.CIFS_PASSWORD and then check if the backup target works well. Thanks.
like:
#!/bin/bash

USERNAME=${Username of SMB/CIFS Server}
PASSWORD=${Password of SMB/CIFS Server}

CIFS_USERNAME=`echo -n ${USERNAME} | base64`
CIFS_PASSWORD=`echo -n ${PASSWORD} | base64`

cat <<EOF >>cifs_secret.yml
apiVersion: v1
kind: Secret
metadata:
  name: cifs-secret
  namespace: longhorn-system
type: Opaque
data:
  CIFS_USERNAME: ${CIFS_USERNAME}
  CIFS_PASSWORD: ${CIFS_PASSWORD}
EOF

kubectl apply -f cifs_secret.yml

Hi @ZanderPittari,
Please replace the data.username to data.CIFS_USERNAME and data.password to data.CIFS_PASSWORD and then check if the backup target works well. Thanks.
like:
#!/bin/bash

USERNAME=${Username of SMB/CIFS Server}
PASSWORD=${Password of SMB/CIFS Server}

CIFS_USERNAME=`echo -n ${USERNAME} | base64`
CIFS_PASSWORD=`echo -n ${PASSWORD} | base64`

cat <<EOF >>cifs_secret.yml
apiVersion: v1
kind: Secret
metadata:
  name: cifs-secret
  namespace: longhorn-system
type: Opaque
data:
  CIFS_USERNAME: ${CIFS_USERNAME}
  CIFS_PASSWORD: ${CIFS_PASSWORD}
EOF

kubectl apply -f cifs_secret.yml

You are a legend, it's backing up now. Didn't realise it had to use a specific naming convention


When #285 is merged we need to add more Goss tests

Now the only HEALTHCHECK test in goss is the Welcome page.
We should add more checks like API and searches.

Testing the API and searches might need database seed (just a friendly reminder).
Something important to be tested is if statics are being properly served: testing some endpoints such as /static/app.js, /static/dashboard.css and /static/admin/css/base.css for example ; )


Add files via upload

@BayerA10G First stage of verification passed, official domain for Legacy Christian Academy is lcamn.org as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@BayerA10G Thank you for your patience while waiting for a review and processing your request. Pull request merged.
Please start requesting the licenses in about an hour, to let the changes to propagate through our system.

Thank you!-ABOn Nov 19, 2024, at 3:39 AM, Philip Torchinsky @.***> wrote:
@BayerA10G Thank you for your patience while waiting for a review and processing your request. Pull request merged.
Please start requesting the licenses in about an hour, to let the changes to propagate through our system.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>


Fix build problem with JRuby

4.49s$ rspec
LoadError: load error: ostruct -- java.lang.StackOverflowError: null
          require at org/jruby/RubyKernel.java:1071
          require at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
           (root) at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/json/generic_object.rb:1
          require at org/jruby/RubyKernel.java:1071
          require at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
           (root) at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/json/common.rb:1
          require at org/jruby/RubyKernel.java:1071
          require at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
           (root) at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/json/common.rb:2
          require at org/jruby/RubyKernel.java:1071
          require at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
           (root) at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/json.rb:1
          require at org/jruby/RubyKernel.java:1071
          require at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
           (root) at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/json.rb:1
          require at org/jruby/RubyKernel.java:1071
          require at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
           (root) at /home/travis/build/snowplow/snowplow-ruby-tracker/lib/snowplow-tracker/payload.rb:1
          require at org/jruby/RubyKernel.java:1071
          require at /home/travis/.rvm/rubies/jruby-1.7.19-d19/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:54
           (root) at /home/travis/build/snowplow/snowplow-ruby-tracker/lib/snowplow-tracker/payload.rb:17
             load at org/jruby/RubyKernel.java:1087
           (root) at /home/travis/build/snowplow/snowplow-ruby-tracker/lib/snowplow-tracker.rb:1
             each at org/jruby/RubyArray.java:1613
           (root) at /home/travis/build/snowplow/snowplow-ruby-tracker/lib/snowplow-tracker.rb:18
           (root) at /home/travis/build/snowplow/snowplow-ruby-tracker/spec/spec_helper.rb:1
           (root) at /home/travis/build/snowplow/snowplow-ruby-tracker/spec/spec_helper.rb:17
           (root) at /home/travis/build/snowplow/snowplow-ruby-tracker/spec/integration/integration_spec.rb:1
           (root) at /home/travis/build/snowplow/snowplow-ruby-tracker/spec/integration/integration_spec.rb:16
           (root) at /home/travis/.rvm/gems/jruby-1.7.19-d19/gems/rspec-core-3.3.2/lib/rspec/core/configuration.rb:1
             load at org/jruby/RubyKernel.java:1087
  load_spec_files at /home/travis/.rvm/gems/jruby-1.7.19-d19/gems/rspec-core-3.3.2/lib/rspec/core/configuration.rb:1327
             eval at org/jruby/RubyKernel.java:1107
           (root) at /home/travis/.rvm/gems/jruby-1.7.19-d19/bin/jruby_executable_hooks:15
The command "rspec" exited with 1.

It seems that we can prevent this by including Contracts inside classes rather than at the outermost level.

The problem here is that the gemspec specifies "~> 0.3" rather than "~> 0.3.0" for the contracts version. This meant that Bundler always installs the latest 0.x version.
So when an incompatible version of Contracts (0.11.0) was released, Travis started to break.

If we don't make the change of moving include Contracts inside classes, the above error occurs for these versions of Contracts:
0.6   0.10   0.11
If we do make the change, an error occurs for these versions of Contracts:
0.3   0.6
So I think we should make the change and alter the gemspec to only be compatible with versions other 0.4, 05, and 0.7+.

I don't see how to support two separate ranges, so instead I will just use "~> 0.7"

Makes sense...


Avoid unused import warning

Getting rid of an unused import.
My rustc version is 1.38.0

Good catch, however this import is required on Linux/BSD, so please, just add a cfg guard to it (you can find the right one a few lines above).

The minimum supported version of Alacritty is Rust 1.36.0 now.
https://github.com/jwilm/alacritty/blob/8a263414572d8ed4d771461de2fdbd9f7e549a75/CONTRIBUTING.md#patches--pull-requests


Curation for VCS of NPM packages with broken submodules

Please have a look at the individual commit messages for the details.

This requires https://github.com/oss-review-toolkit/ort-config/pull/180, I'll rebase once that other PR is merged.


Cannot read property 'ansi_colors' of undefined

The page load the jQuery lib first.
I got the error when i hover the terminal tab.

Do you load two versions of jQuery and unix_formatting.js ?

Do you have that page somewhere online, if not then maybe you can create codepen or jsfiddle demo that show that error?

I found that laravel-tracy library it seems it work without error.

@jcubic yes, i loaded two jquery with same version.

my project is laravel and the package is laravel-tracy.
i found that the terminal will showthe  error if the page already load jquery

I saved $.terminal that is used in terminal variable in unix_formatting.js maybe that will fix your issue, I've tired to reproduce your error by adding another jQuery library after initialization of the terminal and unix formatting is working (I've loaded old version), don't know why it should throw exception.
Can you test new version of unix_formatting.js?

Ah, I know the reason, because both unix_formatting and jQuery terminal use local copy of jQuery that was used when you included the files,
If you use code like this:
<script src="js/jquery-1.8.0.js"></script>
<script src="../repo/js/jquery.terminal-src.js"></script>
<script src="../repo/js/unix_formatting.js"></script>
<script>
$('<div/>').appendTo('body').terminal(function(command, term) {
});
</script>
<script src="js/jquery-1.8.0.js"></script>

everything will work as expected. Do you have different order of scripts?

Can you create demo page with just the front-end files? Maybe on jsfiddle or codepen.

Closing because of no response from OP after two weeks.


Move fuzzing to it's own workflow

This pull request includes changes to the GitHub Actions workflow files .github/workflows/fuzzing.yml and .github/workflows/main.yml. The changes mainly involve the modification of the fuzzing workflow, including the change of permissions, scheduling, and the use of environment variables instead of inputs. The most important changes are:

.github/workflows/fuzzing.yml: Permissions for the fuzzing workflow have been changed from read to write.
.github/workflows/fuzzing.yml: The trigger for the fuzzing workflow has been changed from a workflow call to a daily schedule at 21:00 UTC and a manual dispatch.
.github/workflows/fuzzing.yml: The use of inputs has been replaced with environment variables. This change affects the platform and architecture parameters used in the workflow, as well as several conditionals and commands within the workflow. [1] [2] [3] [4] [5]
.github/workflows/fuzzing.yml: The repository URL used when pushing updates to the fuzzing corpus has been changed to use the github.repository context instead of a hardcoded repository URL.
.github/workflows/main.yml: The reference to the fuzzing workflow has been removed.

These changes seem to aim at making the fuzzing workflow more autonomous and flexible, by allowing it to run on a schedule or manually, and by using environment variables instead of inputs.

coverage: 81.241%. first build
when pulling 25db1e6a3281d04cca6d52f46b6e8d3881b36fca on Alan-Jowett:fix_ci_cd
into e8de891f30db1799985eab9bae1eccb2849dd505 on iovisor:main.


⚠ Jenkins Plugins Artifactory has degraded performance

In 18a28a1, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 2469 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 658df29 after 12 minutes.


enhance the rake task documentation

cc @aspiers

I hope this is no longer relevant, if not please speak up


Implement FlagEmbeddingReranker for search re-ranking

Description
Added FlagEmbeddingReranker for use with BAAI/bge-reranker as a post nodeprocessor
FlagEmbedding is required to use the module.
Fixes # (issue)
Type of Change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Added new unit/integration tests
[ ] Added new notebook (that tests end-to-end)
[x] I stared at the code and made sure it makes sense

Suggested Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added Google Colab support for the newly added notebooks.
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[x] I ran make format; make lint to appease the lint gods

@ColeMurray could you help add examples for this reranking module

@ColeMurray updated the example: https://github.com/run-llama/llama_index/pull/9382
awesome work


📖 Warn when checks are prone to false negatives

Automated tools normally have some false negatives,
some false positives, or both. However, some scorecard criteria
are especially prone to false negatives (where a
project meets the criterion but the tool says it doesn't).
This commit adds warning text about false negatives for
criteria that are especially prone to false negatives.
In all cases the problem is that there are many ways to
implement the criterion, so while the tool may detect some
cases, there are countless other situations it will fail to detect.
While this doesn't fix the problem, warning the humans
will encourage them to double-check these criteria before
making decisions. Sometimes this is the best you can do, and
it's better than not having a warning.
Signed-off-by: David A. Wheeler<EMAIL_ADDRESS>

@olivekl FYI


v1.0.0-rc4

Bumped version. Updated dependencies.
/cc @DylanPiercey

Bithound Code is complaining about duplicated code in the tests. Will be cleaned up before 1.0 release.


Inquiry About Commercial Licensing for Deployed Version

Hello,
I've noticed that the repository on GitHub seems to differ from the version deployed at https://www.morphic.sh/. Are there any plans to offer a commercial licensing plan? I would be very interested in purchasing a commercial license to use the deployed version.
Thank you for your attention to this matter.

Hi @aaronpeng2020
Thank you for your inquiry. It's a very interesting proposal. I had never considered licensing.
However, I'm focused on the development of the main project, so I won't be able to accommodate that.
Currently, there are many authentication products available, and it's easier to integrate them than before.
Please give it a try.

Hi @aaronpeng2020
Thank you for your inquiry. It's a very interesting proposal. I had never considered licensing. However, I'm focused on the development of the main project, so I won't be able to accommodate that. Currently, there are many authentication products available, and it's easier to integrate them than before. Please give it a try.

No, what I meant is that I would like to use the project you developed for commercial purposes, and I would like to obtain your paid license.
I don't need you to do anything extra; I just hope that the features of the commercial version can be the same as those on https://www.morphic.sh/.
Every developer needs to make a living, and I hope there is a way to financially support your continued development.

I think it should be clear that the hosted version (morphic.sh) is not the same version open-sourced here.

We don't plan to add authentication.
reason: https://github.com/miurla/morphic?tab=readme-ov-file#morphic


Block returning from Start until actually started

If you call Stop (or Restart) quickly after calling Start, FSEventStreamInvalidate can be called before FSEventStreamScheduleWithRunLoop. This is an error:
(FSEvents.framework) streamRef->cfRunLoopSourceRef != NULL || streamRef->event_source != NULL(): failed assertion: Must call FSEventStreamScheduleWithRunLoop() before calling FSEventStreamInvalidate()

Instead, Start should block until immediately before it will block on the run-loop.

:+1: looks good to me


[BUG]: [QA | Admit case pop up closes automatically after few sec]

To Reproduce
Steps to reproduce the behavior:

Go to 'Judge user role'
Click on 'Register case-> issue order -> issue notice -> admit case'
See error

Expected behavior
Admit case pop up should not close automatically
Screenshots
admit case.webm

hi @iknoorkaur, the required changes have been made in develop branch. please check this when changes are deployed in QA. cc: @Ramu-kandimalla

Hi @ashishbeehyv, earlier admit case pop up screen was closing automatically but now the below screen (after admitting the case) closes automatically:

Attaching video for reference:
schedule admission hearing.webm
cc: @Ramu-kandimalla @rajeshcherukumalli

Tested, working fine in uat
Attaching jam video for reference: https://jam.dev/c/f4ffc284-6233-4f7c-add5-0eaa16e3f0b3


build and install instructions

I tried to build this and I get
$ npm install
npm WARN deprecated<EMAIL_ADDRESS>This package is deprecated in favor of @types/vscode and vscode-test. For more information please read: https://code.visualstudio.com/updates/v1_36#_splitting-vscode-package-into-typesvscode-and-vscodetest
npm WARN deprecated<EMAIL_ADDRESS>TSLint has been deprecated in favor of ESLint. Please see https://github.com/palantir/tslint/issues/4534 for more information.
npm WARN lifecycle The node binary used for scripts is /Users/redacted/.nodenv/shims/node but npm is using /usr/local/Cellar/node/14.0.0/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

><EMAIL_ADDRESS>postinstall /Users/redacted/Projects/vscode-direnv
> node ./node_modules/vscode/bin/install

Detected VS Code engine version: ^1.44.0
Found minimal version that qualifies engine range: 1.44.0
Fetching vscode.d.ts from: https://raw.githubusercontent.com/Microsoft/vscode/2aae1f26c72891c399f860409176fe435a154b13/src/vs/vscode.d.ts
Error installing vscode.d.ts: TypeError: Cannot set property 'Authorization' of null
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>postinstall: `node ./node_modules/vscode/bin/install`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!      /Users/redacted/.npm/_logs/2020-05-14T17_00_58_982Z-debug.log

Sorry, you blocked the ability to make issues, so I was a left with this.

Yeah, so for anyone following this, the repo is pretty much useless


[#642] Adopt SDK 1.10.0 API changes

Created just to start discussion on changes needed to adopt the new SDK 1.10.0 version.
Closes https://github.com/zcash/secant-android-wallet/issues/642.
This code review checklist is intended to serve as a starting point for the author and reviewer, although it may not be appropriate for all types of changes (e.g. fixing a spelling typo in documentation).  For more in-depth discussion of how we think about code review, please see Code Review Guidelines.
Author

[ ] Self-review: Did you review your own code in GitHub's web interface? Code often looks different when reviewing the diff in a browser, making it easier to spot potential bugs.
[ ] Automated tests: Did you add appropriate automated tests for any code changes?
[ ] Manual tests: Did you update the manual tests as appropriate? While we aim for automated testing of the application, some aspects require manual testing. If you had to manually test something during development of this pull request, write those steps down.
[ ] Code coverage: Did you check the code coverage report for the automated tests?  While we are not looking for perfect coverage, the tool can point out potential cases that have been missed. Code coverage can be generated with: ./gradlew check for Kotlin modules and ./gradlew connectedCheck -PIS_ANDROID_INSTRUMENTATION_TEST_COVERAGE_ENABLED=true for Android modules.
[ ] Documentation: Did you update documentation as appropriate? (e.g README.md, etc.)
[ ] Run the app: Did you run the app and try the changes?
[ ] Screenshots: Did you provide before and after UI screenshots in the description of this pull request?  This is only applicable for changes that modify the UI.
[ ] Rebase and squash: Did you pull in the latest changes from the main branch and squash your commits before assigning a reviewer? Having your code up to date and squashed will make it easier for others to review. Use best judgement when squashing commits, as some changes (such as refactoring) might be easier to review as a separate commit.

Reviewer

[ ] Checklist review: Did you go through the code with the Code Review Guidelines checklist?
[ ] Ad hoc review: Did you perform an ad hoc review?  In addition to a first pass using the code review guidelines, do a second pass using your best judgement and experience which may identify additional questions or comments. Research shows that code review is most effective when done in multiple passes, where reviewers look for different things through each pass.
[ ] Automated tests: Did you review the automated tests?
[ ] Manual tests: Did you review the manual tests?
[ ] How is code coverage affected? We encourage you to compare coverage before and after changes and when possible, leaving it in a better place.
[ ] Documentation: Did you review Docs, README.md, and Architecture.md as appropriate?
[ ] Run the app: Did you run the app and try the changes? While the CI server runs the app to look for build failures or crashes, humans running the app are more likely to notice unexpected log messages, UI inconsistencies, or bad output data. Perform this step last, after verifying the code changes are safe to run locally.

Now we wait with this PR for the SDK 1.10 changes to be merged to the main branch and released, so we can then test it against the production version of our SDK.


Nebula: 401: Unauthorized

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2<IP_ADDRESS> (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region
United States
Description
Can't download any videos off Nebula, despite being able to stream from the site just fine. This issue popped up at least a month ago, and has persisted into the new 6.22 version. I'm not sure exactly when the issue started, as I was in the hospital for a month before coming home to it all of a sudden.
Verbose log
$ yt-dlp -vU --username $(secret-tool lookup username nebula) --password $(secret-tool lookup password nebula) "https://nebula.app/videos/jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr/"
[debug] Command-line config: ['-vU', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://nebula.app/videos/jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr/']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2<IP_ADDRESS> [a86e01e]
[debug] Python version 3.10.4 (CPython 64bit) - Linux-5.15.35-1-lts-x86_64-with-glibc2.35
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.0 (setts), ffprobe 5.0, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.12.0, brotlicffi-<IP_ADDRESS>, certifi-2021.10.08, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.2
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2<IP_ADDRESS>, Current version: 2<IP_ADDRESS>
yt-dlp is up to date (2<IP_ADDRESS>)
[Nebula] Logging in to Nebula with supplied credentials
WARNING: [Nebula] Authentication failed or rejected: HTTP Error 401: Unauthorized
ERROR: [Nebula] jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr: This video is only available for registered users. Use --cookies, --cookies-from-browser, --username and --password, or --netrc (watchnebula) to provide account credentials
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 645, in extract
    self.initialize()
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 551, in initialize
    self._perform_login(username, password)
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/nebula.py", line 148, in _perform_login
    self._nebula_api_token = self._retrieve_nebula_api_token(username, password)
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/nebula.py", line 67, in _retrieve_nebula_api_token
    return self._perform_nebula_auth(username, password)
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/nebula.py", line 37, in _perform_nebula_auth
    self.raise_login_required()
  File "/usr/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1105, in raise_login_required
    raise ExtractorError(msg, expected=True)

May potentially be fixed by https://github.com/yt-dlp/yt-dlp/pull/4002
cc @hheimbuerger

@Jimi-James Sorry, I cannot reproduce this. An error 401 and the "This video is only available for registered users" message usually means that your account credentials are incorrect. Could you please double and triple check that your secret lookup tool is working correctly and the credentials stored in there are absolutely correct (i.e. no additional spaces, etc.)
Maybe try supplying the username/password explicitly on the CLI once for testing. Or try the --netrc mode.
Here's what it looks like for me in the working case (using the release branch):
C:\temp>python -m yt_dlp -vU --username "USER" --password "SECRET" "https://nebula.app/videos/jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr/"
[debug] Command-line config: ['-vU', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://nebula.app/videos/jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr/']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2<IP_ADDRESS> [a86e01e74] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: f92347c31
[debug] Python version 3.10.5 (CPython 64bit) - Windows-10-10.0.19044-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg n4.4-154-g79c114e1b2-20210930 (setts), ffprobe n4.4-154-g79c114e1b2-20210930, phantomjs 2.1.1
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2<IP_ADDRESS>, Current version: 2<IP_ADDRESS>
yt-dlp is up to date (2<IP_ADDRESS>)
[Nebula] Logging in to Nebula with supplied credentials
[Nebula] Authorizing to Nebula
[Nebula] Retrieving Zype access token
[debug] [Nebula] Extracting URL: https://nebula.app/videos/jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr/
[Nebula] jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr: Fetching video meta data
WARNING: [Zype] Login with password is not supported for this website. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/ytdl-org/youtube-dl#how-do-i-pass-cookies-to-youtube-dl  for how to manually pass cookies
[debug] [Zype] Extracting URL: https://player.zype.com/embed/6294cb9fb0d64d0001d39d99.html?access_token=985aaa3c574c5c103095ea5f41f74a4875172ccd7bb302e1007fb13284f2cfc0
[Zype] 6294cb9fb0d64d0001d39d99: Downloading JSON metadata
[Zype] 6294cb9fb0d64d0001d39d99: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 6294cb9fb0d64d0001d39d99: Downloading 1 format(s): hls-4280
[debug] Invoking hlsnative downloader on "https://mf.zype.com/eyJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2d2c20uenlwZS5jb20vNWMxODJkMDY2NDlmMGYxMzRhMDAxNzAzLzYyOTRjYjlmYjBkNjRkMDAwMWQzOWQ5OS82Mjk0Y2I5ZmIwZDY0ZDAwMDFkMzlkOWEvNTQ1YmRhYjE2OTcwMmQwNWJjMDEwMDAwLzUwZjViNWJjLWJiNDctNDM5Ni04Yzc3LWYyMzlhNTAwMDc1Zi5tM3U4IiwicGFyYW1zIjp7IjZwTEtNUTN5IjoiU3FpM05nOTAiLCJGMzhiMWRGNCI6IjYyYjU2ZGNjODZjODk5MDAwMTU1NmFhNyJ9LCJleHAiOjE2NTYwNjgwOTN9.zKkfutq29e5F7YAMAlMKem_DoUb-59LygJeM1K3tefo"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 152
[download] Destination: I Can’t Stop Talking about Resident Evil 4 VR [6294cb9fb0d64d0001d39d99].mp4
[debug] File locking is not supported. Proceeding without locking
[download]  10.5% of ~805.42MiB at   16.77MiB/s ETA 00:54 (frag 16/152)

However, if I'm intentionally supplying an incorrect password (or username), I get an error 400, not 401. Not sure what this means for your issue:
C:\temp>python -m yt_dlp -vU --username "foo" --password "bar" "https://nebula.app/videos/jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr/"
[debug] Command-line config: ['-vU', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://nebula.app/videos/jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr/']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2<IP_ADDRESS> [a86e01e74] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: f92347c31
[debug] Python version 3.10.5 (CPython 64bit) - Windows-10-10.0.19044-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg n4.4-154-g79c114e1b2-20210930 (setts), ffprobe n4.4-154-g79c114e1b2-20210930, phantomjs 2.1.1
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2<IP_ADDRESS>, Current version: 2<IP_ADDRESS>
yt-dlp is up to date (2<IP_ADDRESS>)
[Nebula] Logging in to Nebula with supplied credentials
WARNING: [Nebula] Authentication failed or rejected: HTTP Error 400: Bad Request
ERROR: [Nebula] jacob-geller-i-cant-stop-talking-about-resident-evil-4-vr: This video is only available for registered users. Use --cookies, --cookies-from-browser, --username and --password, or --netrc (watchnebula) to provide account credentials
  File "c:\Develop\yt-dlp\yt_dlp\extractor\common.py", line 645, in extract
    self.initialize()
  File "c:\Develop\yt-dlp\yt_dlp\extractor\common.py", line 551, in initialize
    self._perform_login(username, password)
  File "c:\Develop\yt-dlp\yt_dlp\extractor\nebula.py", line 148, in _perform_login
    self._nebula_api_token = self._retrieve_nebula_api_token(username, password)
  File "c:\Develop\yt-dlp\yt_dlp\extractor\nebula.py", line 67, in _retrieve_nebula_api_token
    return self._perform_nebula_auth(username, password)
  File "c:\Develop\yt-dlp\yt_dlp\extractor\nebula.py", line 37, in _perform_nebula_auth
    self.raise_login_required()
  File "c:\Develop\yt-dlp\yt_dlp\extractor\common.py", line 1105, in raise_login_required
    raise ExtractorError(msg, expected=True)

What if you pass a valid username with wrong password?

What if you pass a valid username with wrong password?

Personally, I'm always getting a 400, even if just the password is wrong. I can reproduce this both by supplying an incorrect password on the command line, as well as by just slightly modifying my .netrc file.
Also, it is consistent with the login page of Nebula's frontend, which also receives a 400 on incorrect password:

I'm not sure how to reproduce OP's error 401.
@Jimi-James Are you sure your Nebula / CuriousityStream subscription hasn't expired? Is your account in good standing?

...I was about to reply saying that my credentials are definitely correct, but suddenly the issue has disappeared. I'm able to download nebula videos again.


Update MD file on new branch 2017-12-03_03-30-42

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Some clarifications about the project

Hi @uPeppe ,
I've read your article on Medium and I have a few questions:

Did you use the project for both Fantacalcio Classic and Mantra?
For the Mantra version, how did you handle players with multiple roles and select the best formation among the various available modules for each matchday?
Did you apply the sin-arcsinh probability distribution to goalkeepers as well?
Which bonuses and penalties did you consider? (e.g., goalkeeper clean sheets, defense modifier)
For each matchday, did you consider validating the results by calculating the delta compared to the actual best formation?

Thanks!

Yes, even though only "normal" role (P-D-C-A) is considered as a feature
Let's say I'd choose the lineup manually, depending on the players expected votes. There is no code for selecting automatically the optimal lineup through optimization method. However, simulate_lineup contains code for calculating the expected points of an input lineup (which is treated as a probability distribution itself) and could potentially be extended to try multiple lineups.
Yes, and for fantavote it results on a distribution skewed to the left (due to the -1 penalty for each goal)
For goalkeepers, only goals -1 and cards if I remember correctly. Clean sheet probability is treated as an additional output, other than vote and fantavote. Defense modifier is not an output for the player prediction, but is taken into account in lineup_simulation as it depends on multiple players. Clean sheet bonus is treated there as well, as not all leagues use that.
Not really in a methodic way. Could be an idea.


Datetime comparison

copy of https://github.com/Unidata/netcdf4-python/issues/639
When a netcdftime datetime object is unable to be compared to another object, the richcmp method raises a TypeError. In previous versions of the module the method would return NotImplemented, causing the appropriate method on the other object to be called and allowing classes to be written which could be compared against netcdftime datetime objects. With the TypeError behaviour this is no longer the case.
Can the previous behaviour be restored?

Seems like simply changing TypeError to NotImplementedError in richcmp would fix this.  Are they any downsides to this?

CC @djkirkham

Seems like simply changing TypeError to NotImplementedError in richcmp would fix this. Are they any downsides to this?

It needs to be return NotImplemented to properly defer to the implemention on the other object, not raise NotImplementedError

This problem has arisen in a project I am working on so I thought to try this and got unexpected results, basically cftime.datetime objects become comparable to anything! ie. breaks not_comparable_4 test in test_richcmp.
Is there anything else that can be done?


#35d8273 breaks FreeBSD as date +%3N is not available

recent commit https://github.com/fisherman/fisherman/commit/35d8273df4717a342bcef81bbfcfa321204fe477#diff-fdd5b8133144df4cb1ef77e46f142452R1752 in #L1852 breaks installation on FreeBSD as there is no %N format available on FreeBSD:

https://www.freebsd.org/cgi/man.cgi?query=date
https://www.freebsd.org/cgi/man.cgi?query=strftime&sektion=3&apropos=0&manpath=FreeBSD+10.3-RELEASE+and+Ports

I simply removed the N and carried on, as I assume this isn't really critical.
dch@wintermute /r/fish-shell> fisher z
switch: Expected exactly one argument, got 2

/usr/local/share/fish/functions/math.fish (line 11):            switch $out
                                                                       ^
in function âmathâ
        called on line 1852 of file ~/.config/fish/functions/fisher.fish
        with parameter list â14634181393N - 0â

in function â__fisher_get_epoch_in_msâ
        called on line 2 of file ~/.config/fish/functions/fisher.fish

in command substitution
        called on line 0 of file ~/.config/fish/functions/fisher.fish

in function âfisherâ
        called on standard input
        with parameter list âzâ

Installing 1 plugin/s
OK Fetch z github.com/fisherman/z
Installing 1 dependency
OK Fetch getopts github.com/fisherman/getopts
switch: Expected exactly one argument, got 7

/usr/local/share/fish/functions/math.fish (line 11):            switch $out
                                                                       ^
in function âmathâ
        called on line 1852 of file ~/.config/fish/functions/fisher.fish
        with parameter list â14634181413N - bc: stdin:1: illegal character: N unexpected<PHONE_NUMBER>3â

in function â__fisher_get_epoch_in_msâ
        called on line 2 of file ~/.config/fish/functions/fisher.fish
        with parameter list âbc: stdin:1: illegal character: N unexpected<PHONE_NUMBER>3â

in command substitution
        called on line 0 of file ~/.config/fish/functions/fisher.fish

in function âfisherâ
        called on standard input
        with parameter list âzâ

Done in 0ms

An alternative might be to re-use the code for /Darwin/ in the preceding case statement, as perl is a reasonable dependency due to git for interactive shell users.

Yep, that's exactly what I was going to suggest :)
Since you are on FreeBSD already, maybe you can send me a patch?

´FreeBSD´ is more ´POSIX´ than most so i suggest reverting the change.


Projections from relay connection queries are not generated

I have created a query using a relay connection response but when I am trying to do the data fetcher tests, I can't found anywhere the query projection root object to generate the query using the builder.
Here is an example:
`
extend type Query {
findSuppliers(findSuppliersInput: FindSuppliersInput!): SupplierConnection
}
type Supplier @connection {
id: ID!
}
`
When I am in the datafetcher test theorically I will need a FindSuppliersProjectionRoot file, but it isn't anywhere, so I don't know how I can do the query in the datafetcher when unit testing it.
I am using the last version of DGS framework and DGS codegen. (4.9.19 and 5.1.16) using Kotlin and Java 11. Spring boot 2.6.3.
Thanks

Also at my org, we do federated graphql introspection to create clients. Code gen at that level will create the appropriate query and projection.
example projection:

public class ItemsProjectionRoot extends BaseProjectionNode {
  public Items_EdgesProjection edges() {
    Items_EdgesProjection projection = new Items_EdgesProjection(this, this);
    getFields().put("edges", projection);
    return projection;
  }

  public Items_PageInfoProjection pageInfo() {
    Items_PageInfoProjection projection = new Items_PageInfoProjection(this, this);
    getFields().put("pageInfo", projection);
    return projection;
  }
}

Thanks @manymotes . Using the introspected schema is a great suggestion to get around this issue.

We don't plan to add support for this due to the dynamic generation of these types. Codegen is designed to work with static schema type definitions. While this is a gap in functionality related to using the client codegen api, there are ways to work around this be either using the introspection schema as described in earlier comments, or by using query strings directly where this feature is being used.

Thanks for the answers
I am currently using the projection with the string and everything is ok until I want to test the content of that object because is treated like a LInkedHashMap
I don't only want to test that the projection/query runs well but also that the content of the edge is equals to the one I have mocked.
It's almost impossible to test it with a natural way, I have tried to map the response into a Connection but it's not possible (is an interface)
Then I have tried to map it to a DefaultConnection but it's impossible because the mapper that is used internally by DGS does not know how to map a Guava ImmutableList class
Probably if I found some way to fix that another problem will rise up.
It's very difficult to test Relay datafetcher responses.
Any ideas? for my test is not OK to just test that the response has a edges or pageInfo field.
Cannot find a deserializer for non-concrete Collection type [collection type; class graphql.com.google.common.collect.ImmutableList, contains [simple type, class graphql.relay.Edge<bff.suppliers.generated.types.Supplier>]]
Thanks !

I have found a solution but is pretty awful to be honest, would be great if the framework could do anything about it in the future because testing relays is a nightmare
        val response = dgsQueryExecutor.executeAndExtractJsonPathAsObject(
            findSuppliersProjection,
            "data.${QUERY.FindSuppliers}",
            object : TypeRef<LinkedHashMap<String, Any>>() {}
        ).block()!!

        val jsonResponse = objectMapper.writeValueAsString(response)
        val jsonNode = objectMapper.readTree(jsonResponse)
        val jsonNodeObject = jsonNode["edges"].first()["node"]
        val jsonObject = objectMapper.writeValueAsString(jsonNodeObject)

        val responseObject = objectMapper.readValue(jsonObject, object : TypeReference<Supplier>() {})

        assertThat(responseObject).isEqualTo(supplierConnection.edges.first().node)

Using codegen on introspected schemas looks like a good option, but is that the strategy implied https://netflix.github.io/dgs/generating-code-from-schema/#generating-client-apis? Introspection is not mentioned in the rest of the page, but it does mention relay and connections.

@MayCXC - I just learned about this option from this discussion. I will add it to the documentation so it's available as an option.

@zanonena - what if you created a data model for the response with all the fields by hand? We added the @connection feature for dynamically generating the edges and connection objects intending it to be used standalone. Codegen plugin will not work with it because it uses static schema. I think you will just need to create data model so you can deserialize using that class from the response.

@srinivasankavitha
Yes I finally used my own SupplierConnection type and I will not be using the @connection directive
Thank you


Fix tags and add button to control subs in firefox

Firefox  does not have in his controls player as turn on or off the captions.

Looks good m8


[WIP] python virtualenv operator

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
This makes an operator that allows simple Python code to execute simple virtualenvs. This is a WIP. I will probably change it to a Mixin or restructure so we can write operators with execute that have different requirements than the base airflow ones
JIRA

[ ] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-XXX

Description

[ ] Here are some details about my PR, including screenshots of any UI changes:

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #2446 into master will decrease coverage by 0.26%.
The diff coverage is 31.42%.

@@            Coverage Diff             @@
##           master    #2446      +/-   ##
==========================================
- Coverage    69.3%   69.04%   -0.27%
==========================================
  Files         146      146
  Lines       11244    11313      +69
==========================================
+ Hits         7793     7811      +18
- Misses       3451     3502      +51

Impacted Files
Coverage Δ

airflow/operators/python_operator.py
59.83% <31.42%> (-38.28%)
:arrow_down:

airflow/jobs.py
75.05% <0%> (-0.41%)
:arrow_down:

airflow/models.py
87.33% <0%> (+0.04%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 28aeed4...8f7ab68. Read the comment docs.

I like the idea, please provide a docstring and add to the Sphinx autodocs. Also unit test will be needed.

cc: @mistercrunch @aoen @bolkedebruin
Ready for review

How to about virtualenvwrapper and pyenv?
They allow you to reuse preconfigured environments.

This doesn't look too useful because in most cases you probably won't keep around all virtualenvs because of disk usage. With wheel caching, virtualenvs can actually install pretty fast. (for example, airflow cached, with 100 deps, takes a few seconds)

LGTM


[Tabs] Change the default indicatorColor and textColor prop values to "primary"

Breaking changes

[Tabs] Change the default indicatorColor and textColor  prop values to "primary".
This is done to match the most common use cases with Material Design.
-<Tabs />
+<Tabs indicatorColor="primary" textColor="inherit" />

Fix #24286
Improve the demos in the documentation: https://deploy-preview-25063--material-ui.netlify.app/components/tabs/

I have push changes to improve the demos of the tabs; I have been keen on doing them for a long long time

Nice! The changes look good :]

@Dripcoding Thanks, a great step forward for the tabs components.


Add memory pool configuration to datafusion-cli

Is your feature request related to a problem or challenge?
While trying to test https://github.com/apache/arrow-datafusion/pull/7400 with datafusion-cli I found I can't do it with datafuson-cli because datafusion-cli doesn't have a memory manager enabled.
Describe the solution you'd like
I would like to add two new new command line options to datafusion-cli

-m / --mem-limit that if set, would set the memory pool size limit. If unset no memory pool is used
--mem-pool-type=<greedy|fair>, defaults to greedy that specifies the pool type: GreedyMemoryPool or FairSpillPool respectively

Examples of usage
# memory is not limited
datafusion-cli -c 'select 1, 2 from foo';

# run query with greedy memory pool set to use 10G
datafusion-cli --memory-limit 10G -c 'select 1, 2 from foo';

# run query with greedy memory pool set to use 10G
datafusion-cli -m 10G -c 'select 1, 2 from foo';

# run query with fair memory pool set to use 10G
datafusion-cli --pool-type=fair -m 10G -c 'select 1, 2 from foo';

See https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.FairSpillPool.html for more details
Describe alternatives you've considered
I also thought about setting the pools via SET commands (like setting the target batch size). However,  I don't think we should allow change memory limits via SQL because  memory limits is likely not something a multi-tenant system would like to do . It should be setup before the session starts or by the runtime system, not the user in SQL
Additional context
Since this is well specified and is mostly an exercise in figuring out how datafusion-cli works, I think this would make a good first project

I am glad to pick this ticket up this weekend.


ERROR Unexpected exception: 'i' format requires -2147483648 <= number <=<PHONE_NUMBER>

While training, at the start of the BPE processing, I got this error:
ERROR Unexpected exception: 'i' format requires -2147483648 <= number <=<PHONE_NUMBER>
On a new machine, with Ubuntu 20 and the 30XX config, I tried many times to reinstall with various conditions, mainly:

virtualenv/conda
python3.6/3.8
requirements versions
local redefinition (en-US + UTF-8)
etc...

I finally succeeded in making this working without error. But, having done a lot of experiments, I do not know what was really the change that made it running properly.
I'm now back on an old machine, with Ubuntu 18 and 1080 config, that was historically working properly. Something should have been changed on its config (system update?), because I also get this error. After many reinstall experiments, I do not succeed in making it working again.
Any idea on the problem, and what should be done to solve it?
PS: no error in the LOG file. Just the above error message on the main terminal screen.

Bad news! The Docker install is also crashing with this error!
I will now try to upgrade from Ubuntu 18 to 20. Hoping this will solve some bad lib version conflicts somewhere.
:(

I just lost between one and two full weeks on this problem, with projects and clients waiting for it.
It's typically what is a so-called critical blocking bug!
If someone could have a look at it and help me, it would be nice.

YESSSSS !!!!
Not sure the following is sufficient, nor exhaustive, but this seems to solved the problem for me:

upgrading from Ubuntu 18 to 20
re-installing MMT using the 30XX procedure

The official Docker installation is still not working.
I think the 30XX config should now be validated as the new official release, and a new docker image built with it.

YESSSSS !!!! Not sure the following is sufficient, nor exhaustive, but this seems to solved the problem for me:
* upgrading from Ubuntu 18 to 20

* re-installing MMT using the 30XX procedure

The official Docker installation is still not working.
I think the 30XX config should now be validated as the new official release, and a new docker image built with it.
PS: in my new re-install, I used a virtualenv config with Python3.8, and #601

Is there still someone in the MMT teams, with a will to make the tool evolve?
The growing silence, week after week, is disturbing.


Freedesktop bootloader spec

Looks like we need to implement it.
https://www.freedesktop.org/wiki/Specifications/BootLoaderSpec/

I will look at it - thanks!

Easy to implement. Not security at all but that's how it is


appending a base tag to the head before jasmine tests run.

fixes #667

@codydaig @mleanos @ilanbiala code review?

LGTM. Doesn't effect the current tests in any way, and all tests pass regardless but I think it's a good addition for any future client side tests.

Yeah, doesn't really impact anything right now, but I needed it for creating some of these client tests.

@cdriscol Those tests are looking good!

@cdriscol it would be a great addition if you could add client side tests :)

@lirantal, I plan on adding all my current client side tests and more once I've had some time to refactor a bit.
You can assign #663 to me if you'd like.. I don't think I can assign it to myself..?

@cdriscol sure, let's take it in #663

@cdriscol when linting the code it reports the following issue:

Can you advise?

@cdriscol actually when re-running it after an npm install I don't see the problem anymore. I'm not sure if it's something I missed or an actual problem, can you look into it just so we're on the safe side?

@lirantal - did a clean npm install and jshint passes with no errors.
I think the only way you'd get that not defined error is if your .jshintrc didn't have "jasmine": true or if your jshint was out of date (i.e. before they added jasmine 2.1 globals)

@cdriscol got it, thanks. do we have the correct version now on 0.4.0 branch for jasmine on package.json ?

@lirantal - yeah, all of our current versions should be good in our package.json.


Contributions ?

Hello I was thinking of building a project similar to this myself but came across yours. Are you accepting contributions assuming I find the time ? I ask because I did not see or notice any guidelines on contributing.

Of course that is welcome. If you want to do bug fixes or implement a small improvement, just open a PR. If you have a big feature plan, it is better to first open an issue to talk about it.


Running tensorflow backend on single gpu but on multi gpu machine

This result with:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.39                 Driver Version: 375.39                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 8CFB:00:00.0     Off |                    0 |
| N/A   58C    P0    58W / 149W |  10873MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 9657:00:00.0     Off |                    0 |
| N/A   74C    P0    65W / 149W |  10873MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | A78F:00:00.0     Off |                    0 |
| N/A   42C    P0    71W / 149W |  10871MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           Off  | BB06:00:00.0     Off |                    0 |
| N/A   67C    P0   141W / 149W |  10941MiB / 11439MiB |     58%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      4543    C   python                                       10869MiB |
|    1      4543    C   python                                       10869MiB |
|    2      4543    C   python                                       10867MiB |
|    3      4543    C   python                                       10937MiB |
+-----------------------------------------------------------------------------+

As can be seen above: Only 1 gpu is being used. But the memory has been allocated for all 4 gpu
Any idea how to prevent it?

This is default behavior for TensorFlow
If you want TensorFlow to not allocate memory for "all of the GPUs", add the following to your script:
import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"
This will make it so that only the 0th GPU is visible to TensorFlow.
In your case, you can choose any in the range [0, 3].
If you wanted, for example, the 0th and 2nd GPUs to both be visible, replace "0" with "0,2".
If you want TensorFlow to not allocate "all of the memory" for the GPUs visible to it, then add:
from keras import backend as K
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)
K.set_session(sess)
Hope this helps!

Hey thanks for the information. I'll check it out. While this is not tensorflow board, can you still explain why does it allocate all GPU even if it uses only a single one?

While TensorFlow utilizes only one GPU by default, I believe it allocates the full memory of all of the GPUs such that the user can readily [that is, front loading the setup time] use the TensorFlow interface to incorporate use of the other GPUs in their training/inference pipelines.
Sorry if that's a bit of a non-answer, but maybe these links will help:
TensorFlow notes on device placement with GPUs
Keras blog notes on multi-GPU training

Thanks!

I am having the same issue. I am using Keras R version, do you know to do this in R version?

Set the environment variable in your shell to
CUDA_VISIBLE_DEVICES=0

import os
os.environ["CUDA_VISIBLE_DEVICES"]="0,1,2"

config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)
keras.backend.set_session(sess)

def _get_available_devices():
    from tensorflow.python.client import device_lib
    local_device_protos = device_lib.list_local_devices()
    return [x.name for x in local_device_protos]

print _get_available_devices()
```python

Results in
u'/device:CPU:0', u'/device:GPU:0'

The CUDA_VISIBLE_DEVICES solution ignores the possibility that one is running stuff on a queue, where one cannot know at runtime which device will be used! Has a better solution appeared in the two years that tensorflow already exists?

Same issue, But I want to know, how to release the rest gpus if tf session has started, without killing process and restarting it , thank you!

Same question, how to set a GPU per python thread. An environment variable is nice for setting the visdisle GPU for the whole script, but what if I have two threads in the same script, consuming a queue, each controlling their own GPU?

set a GPU per python thread

Same question here!!! I have google a lot but have not found any good solution. Any help will be greatly appreciated!

set a GPU per python thread

Same question here!!! I have google a lot but have not found any good solution. Any help will be greatly appreciated!


CI/CD: int/main: lint, build, pack, and upload to Azure Blob int/main instances; remove dependency on ts-node

Completed