WCS WebCoverageService authentication support?

Hello,  is there some way to include authentication  in the WebCoverageService class, like username and password parameters as  in WebMapService_1_1_1?
Thanks
Alessandro

Hi. Did you find the answer?
Thanks

You can now add an Authentication object as follows:
from owslib.wcs import WebCoverageService
w = WebCoverageService('https://example.org/wcs', Authentication(username='foo', password='bar')

@tomkralidis  Unfortunately the website I'm trying to connect is not working . I check your solution after that.

Here is the completed code (thanks to @tomkralidis):
from owslib.util import Authentication
from owslib.wcs import WebCoverageService
wcs = WebCoverageService('http://maps.isric.org/mapserv?map=/map/phh2o.map', auth=Authentication(username='foo', password='bar'),
                         version='1.0.0')

bbox = (-1784000, 1356000, -1140000, 1863000)
response = wcs.getCoverage(
    identifier='phh2o_0-5cm_mean',
    crs='urn:ogc:def:crs:EPSG::152160',
    bbox=bbox,
    resx=250, resy=250,
    format='GEOTIFF_16')
with open('./data/Senegal_pH_0-5_mean.tif', 'wb') as file:
    file.write(response.read())

import rasterio
CEC = rasterio.open("./data/Senegal_pH_0-5_mean.tif", driver="GTiff")

from rasterio import plot
plot.show(CEC, title='Mean pH between 0 and 5 cm deep in Senegal', cmap='gist_ncar')

1 problem:

It gives me this error (do to the server errors):
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='maps.isric.org', port=80): Read timed out. (read timeout=30)
For now, I cannot create a username and password and check if this works or not.

The servers work just fine for now. I have a new problem and would appreciate any help from you,
Following this code, my problem is with Authentication procedure. I add an Authentication object to the WebCoverageService object. So, I did as required. Here is the working code with the repaired servers:
from owslib.util import Authentication
from owslib.wcs import WebCoverageService
wcs = WebCoverageService('https://maps.isric.org/mapserv?map=/map/phh2o.map', auth=Authentication(username='foo', password='bar'),
                         version='1.0.0')

bbox = (-1784000, 1356000, -1140000, 1863000)
response = wcs.getCoverage(
    identifier='phh2o_0-5cm_mean',
    crs='urn:ogc:def:crs:EPSG::152160',
    bbox=bbox,
    resx=250, resy=250,
    format='GEOTIFF_16')
with open('./data/Senegal_pH_0-5_mean.tif', 'wb') as file:
    file.write(response.read())

import rasterio
CEC = rasterio.open("./data/Senegal_pH_0-5_mean.tif", driver="GTiff")

from rasterio import plot
plot.show(CEC, title='Mean pH between 0 and 5 cm deep in Senegal', cmap='gist_ncar')

However, I could not find a way to create a username and password in the https://maps.isric.org/ web service so that I can pass them to the Authentication object I created. Could you please tell me how to do this?

Here is the completed code (thanks to @tomkralidis):
from owslib.util import Authentication
from owslib.wcs import WebCoverageService
wcs = WebCoverageService('http://maps.isric.org/mapserv?map=/map/phh2o.map', auth=Authentication(username='foo', password='bar'),
                         version='1.0.0')

bbox = (-1784000, 1356000, -1140000, 1863000)
response = wcs.getCoverage(
    identifier='phh2o_0-5cm_mean',
    crs='urn:ogc:def:crs:EPSG::152160',
    bbox=bbox,
    resx=250, resy=250,
    format='GEOTIFF_16')
with open('./data/Senegal_pH_0-5_mean.tif', 'wb') as file:
    file.write(response.read())

import rasterio
CEC = rasterio.open("./data/Senegal_pH_0-5_mean.tif", driver="GTiff")

from rasterio import plot
plot.show(CEC, title='Mean pH between 0 and 5 cm deep in Senegal', cmap='gist_ncar')

1 problem:

It gives me this error (do to the server errors):
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='maps.isric.org', port=80): Read timed out. (read timeout=30)
For now, I cannot create a username and password and check if @tomkralidis's solution works or not.

The servers work just fine for now. I have a new problem and would appreciate any help from you,
Following this code, my problem is with Authentication procedure. I add an Authentication object to the WebCoverageService object. So, I did as required. Here is the working code with the repaired servers:
from owslib.util import Authentication
from owslib.wcs import WebCoverageService
wcs = WebCoverageService('https://maps.isric.org/mapserv?map=/map/phh2o.map', auth=Authentication(username='foo', password='bar'),
version='1.0.0')
bbox = (-1784000, 1356000, -1140000, 1863000)
response = wcs.getCoverage(
identifier='phh2o_0-5cm_mean',
crs='urn:ogc:def:crs:EPSG::152160',
bbox=bbox,
resx=250, resy=250,
format='GEOTIFF_16')
with open('./data/Senegal_pH_0-5_mean.tif', 'wb') as file:
file.write(response.read())
import rasterio
CEC = rasterio.open("./data/Senegal_pH_0-5_mean.tif", driver="GTiff")
from rasterio import plot
plot.show(CEC, title='Mean pH between 0 and 5 cm deep in Senegal', cmap='gist_ncar')
However, I could not find a way to create a username and password in the https://maps.isric.org/ web service so that I can pass them to the Authentication object I created. Could you please tell me how to do this?

However, I could not find a way to create a username and password in the https://maps.isric.org/ web service so that I can pass them to the Authentication object I created. Could you please tell me how to do this?

@amirmohammadmir glad you got this working.  Your question on creating a username/password is outside the scope of OWSLib.  Having said this, I believe @jorgejesus may have an an idea as I think he works on the ISRIC platform.

However, I could not find a way to create a username and password in the https://maps.isric.org/ web service so that I can pass them to the Authentication object I created. Could you please tell me how to do this?

@amirmohammadmir glad you got this working. Your question on creating a username/password is outside the scope of OWSLib. Having said this, I believe @jorgejesus may have an an idea as I think he works on the ISRIC platform.

Dear @tomkralidis . Thank you for suggesting @jorgejesus. Fortunately, I managed to run the code and get the outputs. Here is the revised version:
from owslib.wcs import WebCoverageService
wcs = WebCoverageService('https://maps.isric.org/mapserv?map=/map/phh2o.map',
                         version='1.0.0')

bbox = (-1784000, 1356000, -1140000, 1863000)
response = wcs.getCoverage(
    identifier='phh2o_0-5cm_mean',
    crs='urn:ogc:def:crs:EPSG::152160',
    bbox=bbox,
    resx=250, resy=250,
    format='GEOTIFF_INT16')
with open('./data/Senegal_pH_0-5_mean.tif', 'wb') as file:
    file.write(response.read())

import rasterio
CEC = rasterio.open("./data/Senegal_pH_0-5_mean.tif", driver="GTiff")

from rasterio import plot
plot.show(CEC, title='Mean pH between 0 and 5 cm deep in Senegal', cmap='gist_ncar')


Add Basic "Down for Maintenance" Message

We need to add a simple method for disabling user logins to ECMPS 2.0 (driven by downtime that will be necessary during the transition to ICAM for authentication).
A more robust version of this will be implemented later (see #6243)
We should add a new environment variable (EASEY_AUTH_API_DISABLE_LOGIN) for the auth-api app.
If EASEY_AUTH_API_DISABLE_LOGIN is set to true, display the message "ECMPS is currently down for maintenance." instead of the normal "Log In" form:

Acceptance criteria:

[x] If EASEY_AUTH_API_DISABLE_LOGIN is set to false, or is not set at all, the normal "Log In" form should display on the ECMPS 2.0 home page.
[x] If EASEY_AUTH_API_DISABLE_LOGIN is set to true, the message "ECMPS is currently down for maintenance." should display instead of the normal "Log In" form

FYI - @mark-hayward-erg @spetros-do
Unable to verify in test. The EASEY_AUTH_API_DISABLE_LOGIN is missing from the test env, however, I do see it in the dev env.
Steps:

Login to cloud.gov, target the easey-org and test space.
View the auth-api's environmental variables (cf env auth-api)
The EASEY_AUTH_API_DISABLE_LOGIN is missing from the User-Provided env variables list.

@ibarra-michelle @mark-hayward-erg
We plan to implement same solution to test env.

Verified in test, pending text change. I was able to set the EASEY_AUTH_API_DISABLE_LOGIN to true, restage the app, clicked the "Log in" button, then and I received the notification, "ECMPS 2.0 Login is disabled due to Maintenance. please contact information for further assistance." (see the screenshot below). I set the EASEY_AUTH_API_DISABLE_LOGIN to false, restaged the app, and was able to successfully login to ECMPS 2.0.
@spetros-do could you modify the disabled message to state, "The ECMPS 2.0 Login has been disabled due to maintenance. Please read the notifications in home page’s "What's New" box and ECMPS Newsletter. For further assistance, please contact us.”


WIP examples

Closing mostly because it was a bit difficult to decide which parts of our various schema definitions needed examples, as well as due to problems with the way SwaggerParser resolved references to the example file (Would work for nested schemas up until the point where you refered to the parent, at which point the parser would "dedudpe" the nested schema leaving it exmpe less (i.e. Aliases wouldn't have an example but items would (including the aliases).


STARsolo 2.7.5b (Tutorial Feedback #989 Oct 29th 2020)

looks like there are two independent issues here:
(ping @jennaj @mtekman @blankenberg )

Yes, Homo_sapiens.GRCh37.75.gtf is not correctly auto detected as 'gtf' when uploaded from zenodo. I have seen this before (on other occasions), and it needs a fix in the sniffer (probably more than just a 'paper cut'?)
STARsolo 2.7.5b (the current default on usegalaxy.eu)  is broken (or need a different kind of transcripts annotation file?). The same inputs work with a previous tool version (e.g.: 2.7.2b1 )

STARsolo 2.7.5b (the current default on usegalaxy.eu) is broken (or need a different kind of transcripts annotation file?). The same inputs work with a previous tool version (e.g.: 2.7.2b1 )

Ack! Looks like I need to add more tests...

Some useful links:

The GFF3 sniffer is defined as 9 tab-seperated fields, and its test dataset reflects that.

The GTF sniffer is defined as at least 9 tab-seperated fields (with the first 8 matching the GFF format), with the last Group field containing semicolon pairs, and its test dataset has that.

So from what I can see, it's the 9th column which is the deciding factor.
If we look at Homo_sapiens.GRCh37.75.gtf given by Zenodo, we see:
chr1	pseudogene	gene	11869	14412	.	+	.	gene_id "ENSG00000223972"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene";
chr1	processed_transcript	transcript	11869	14409	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000456328"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-002"; transcript_source "havana";
chr1	processed_transcript	exon	11869	12227	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000456328"; exon_number "1"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-002"; transcript_source "havana"; exon_id "ENSE00002234944";
chr1	processed_transcript	exon	12613	12721	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000456328"; exon_number "2"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-002"; transcript_source "havana"; exon_id "ENSE00003582793";
chr1	processed_transcript	exon	13221	14409	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000456328"; exon_number "3"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-002"; transcript_source "havana"; exon_id "ENSE00002312635";
chr1	transcribed_unprocessed_pseudogene	transcript	11872	14412	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000515242"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-201"; transcript_source "ensembl";
chr1	transcribed_unprocessed_pseudogene	exon	11872	12227	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000515242"; exon_number "1"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-201"; transcript_source "ensembl"; exon_id "ENSE00002234632";
chr1	transcribed_unprocessed_pseudogene	exon	12613	12721	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000515242"; exon_number "2"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-201"; transcript_source "ensembl"; exon_id "ENSE00003608237";
chr1	transcribed_unprocessed_pseudogene	exon	13225	14412	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000515242"; exon_number "3"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-201"; transcript_source "ensembl"; exon_id "ENSE00002306041";
chr1	transcribed_unprocessed_pseudogene	transcript	11874	14409	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000518655"; gene_name "DDX11L1"; gene_source "ensembl_havana"; gene_biotype "pseudogene"; transcript_name "DDX11L1-202"; transcript_source "ensembl";

So the 9th column, should make this a GTF file and not a GFF.  Hmm, the sniffer actually might be broken...

Unfortunately, it is a mess

GTF is GFF2 (https://www.ensembl.org/info/website/upload/gff.html)
GFF3 can also contain semicolon separated pairs (http://www.ensembl.org/info/website/upload/gff3.html)
see also: http://gmod.org/wiki/GFF3

to write a correct sniffer is near to impossible, unless you want to include some testing on the second columns (i.e. do we have the term "transcript" present) - but that is tricky as well
the easiest is probably to allow gff, gff3 and gtf as input ?

to write a correct sniffer is near to impossible, unless you want to include some testing on the second columns (i.e. do we have the term "transcript" present) - but that is tricky as well

this is something we could definitely do -- I've seen the 9th column tested for keywords such as "gene_id" and "transcript_id"
                # Check attributes for gene_id, transcript_id
                attributes = parse_gff_attributes(hdr[8])
                if len(attributes) >= 2:
                    if 'gene_id' not in attributes:
                        return False
                    if 'transcript_id' not in attributes:
                        return False

@mtekman Future reference / note to future self

For testing all uploads:
   ./run_tests.sh -integration test/integration -- -k "upload_datatype"

For testing very specific uploads:

in galaxy/test/integration/test_datatype_upload.py change def test_upload_datatype_auto(instance, test_data, temp_file) to:

def test_upload_datatype_auto(instance, test_data, temp_file):
    nam = test_data.datatype.__class__.__name__
    if nam == "Anndata":   # or whatever class you want to test only
        upload_datatype_helper(instance, test_data, temp_file)

to write a correct sniffer is near to impossible, unless you want to include some testing on the second columns (i.e. do we have the term "transcript" present) - but that is tricky as well

this is something we could definitely do -- I've seen the 9th column tested for keywords such as "gene_id" and "transcript_id"
                # Check attributes for gene_id, transcript_id
                attributes = parse_gff_attributes(hdr[8])
                if len(attributes) >= 2:
                    if 'gene_id' not in attributes:
                        return False
                    if 'transcript_id' not in attributes:
                        return False

so it wouldn't be too outlandish to test the second column

The problem is in the code you pasted above, transcript_id is not present in the "gene" lines, so the check for transcript_id fails. For context, this code has been there for 11 years since commit 19a956aedbc8b5e9229ac8b0b8f378e487c547ee , probably following https://genome.ucsc.edu/FAQ/FAQformat.html#format4 (which mandates both gene_id and transcript_id for every non-comment line).
I will open a PR to just test for gene_id .

@nsoranzo oh I just saw your comment now
I have tried something similar with this PR: https://github.com/galaxyproject/galaxy/pull/10588

If we don't require both gene_id and transcript_id for GTF validation (sniffing) ... and put this out as an example in training material for valid GTF format, that is a problem. Mappers may work Ok with a poorly formatted GTF but most downstream tools are very sensitive about content. More tools accept (and require) GTFs than GFF3s (or some hybrid that tend to default to GFF sniffing).
An out-of-specification reference annotation is one of the most common reason, or actually the most common reason, for odd results or outright tool errors. Mismatched genome inputs come in second. Even if sniffing is modified it won't fix the root problems (and there have been discussions about that -- including: recognize common format issues, split up the header vs content at Upload, assign the best-guess datatype to content post-fix and save/hide the header, etc). Both should also be a wrapper-level check imho, before bothering with tool execution and eventual odd-failure, and should gracefully report a meaningful error message directly in stderr within the error dataset(s) if not a common format issue. Point to FAQs or help we integrate (is always better if users are not redirected away from the core application).
Any header line will also cause validation/sniffing to fail. And cause tools to fail if the user changes the datatype (to get the tool to accept the input) but the content is still problematic. The warning about mismatched datatypes vs data content is not noticed, or rather, likely not understood. Yet many common data providers add in header lines into GFT data for provenance/versioning purposes. Ends up as GFF. User changes to GTF. Mapping usually Ok, downstream tools fail, ask for help, have to start completely over from the start after "fixing" the ref anno.
I know this isn't easy .. but I agree it is more than a papercut. Tool wrappers that consume GFF/GTF/GFF3 need to catch or fix common out-of-specification cases, with a biologist understandable error message if caught-not-auto-fixable.
Is there a reason why tool wrappers cannot strip/ignore header lines in GTFs before submitting the data to tools during execution? If they all did that, we could relax the sniffing. And save users extra steps (remove beginning of file; Select not matching regex ">#" or ">$"; etc).
Same for GFF3 -- extra comment lines at the start are the biggest issue. Blank lines and interspaced comments lines are second. Couldn't both be handled by tool wrappers at runtime?
I think is important that we make Galaxy much easier to use with regard to ref annotation. However achieved. Big content problems --yes, should error, but not with some odd python or other error ("job exceeds memory allocation" or "exceeds runtime" very common). For small format issues that are very common and known by all as gotchas, we should seriously consider what can be done about making those work better in Galaxy, for both practical reasons and to make end-users happier.

@jennaj So you're suggesting we relax the sniffers to be more flexible with the annotation files (which might have header issues), but to also notify the user when the annotation file is not to spec (to save them from downstream issues) -- yep sounds good to me.
I'm not sure stripping headers before tool execution will work though, as I think STAR requires certain header lines to function properly (https://github.com/alexdobin/STAR/issues/988) and I imagine other mappers will too.
For a paper cut, I think we could implement the user notification for out-of-spec files. Beyond that I'm not certain

With https://github.com/galaxyproject/galaxy/pull/10588 (already in 20.09) we now require only gene_id in the attributes column, which allows GTF files from Ensembl to be sniffed.
If this is problematic for some tools, we can create a different "Ensembl GTF" datatype.

Update: See https://github.com/galaxyproject/galaxy/issues/10875
This change impacted a few built-in functions, several tools, and leads to a larger discussion about input validation in general.


Make Flexible Sync Less Painful By Auto-Opting Types Into Syncing

Problem
The new(ish) Flexible Sync feature is...complicated. I understand why it exists, but it's:

Unintuitive
Very messy to set up — nested closures as a function parameter in flexibleSyncConfiguration with braces vomited all over the place.
Very easy to mess up — when I add a new type to the model, I have to remember to go add it to my subscriptions and there's no compiler help to remind me.
If my newly-added type links to other types, I have to remember to add those types (and manually check if I already did add them) or I get the worst possible outcome: a silent failure with no error messages and just nil objects at the end of links when, in fact, those objects aren't nil in the database. (That ALONE should have been disqualifying for shipping this implementation of Flexible Sync because it's essentially silent data corruption, but that ship has apparently sailed and we're stuck with what Mongo did.)

The old Partition Based sync was much cleaner. Especially for cases where developers just want to sync everything.
Solution
Because Partition Based sync is pretty clearly deprecated, I'd like the Swift SDK to offer a much cleaner API to deal with Flexible Sync. This API does one thing: "Sync everything in my database, except these things: X, Y, Z"
If I don't provide a type to exclude, this new API just sets up all the subscriptions I need for every object type I have. That way, I never need to worry about false-dead-ends in linked objects or manually adding a subscription when I create a new model object—it restores Realm to how it worked before this Flexible Sync API asteroid hit.
In short, this new API would be an "opt out" setup rather than an "opt in" setup. That's much more appropriate for the kinds of apps I build with Realm and neatly solves the problems above.
Note: I don't use queries in subscriptions because there are no queries for me to use. I need all the objects of every type in my app. So I just have a list of subscriptions with object types—a list I now need to manually update and track. I understand that the query approach makes sense for many apps, but not in my case.
Alternatives
Partition Based sync, I guess? The website has big warning labels on it though, so I'm sure it's not long for the world.
How important is this improvement for you?
Would be a major improvement
Feature would mainly be used with
Atlas Device Sync

Hi @bdkjones thanks for taking the time for giving some feedback about flexible sync. We're still tuning the API to offer a better developer experience while maintaining speed and performance.
We are working on an extension of the flexible sync API which soon we will release as a preview https://github.com/realm/realm-swift/pull/8244, we'll love some feedback when this is released as well.

@dianaafanador3 Thanks. #8244 doesn't look like it'll help too much. It's become a little irrelevant, I guess, because I've just about had it with Flexible Sync. I spun up a new Atlas App Service, turned on Flexible Sync with developer mode enabled, ensured that the default Role was set to always allow read/write access and when I try to do this very simple thing:
try! realm.write({
    realm.add(newFoo)
})

I get this garbage:

It's a configuration issue, I'm sure. But the backend of MongoDB is such a blackhole of pane after pane after pane after pane of pain, I have no idea. I'm not querying a field. The role permissions are set. I opened the realm like this:
let config = app.currentUser!.flexibleSyncConfiguration { subs in
            subs.append(QuerySubscription<Foo>(name: "foo"))
        }

Realm.asyncOpen(configuration: config, callbackQueue: .main) { result in
    ...
}

I'm tired and I'm frustrated and I want to throw MongoDB out and start all over with a different service. I'm guessing that the "feedback period" that Mongo ran on Flexible Sync last year didn't really include a broad swath of real-world developers because we're all very busy and can't take production apps to new, untested APIs. Mongo didn't get a lot of honest feedback about this thing and, as such, it shipped a very complex, fragile, cumbersome solution to a pretty simple problem: "Let developers tell us what subset of their data on the server they want to sync to devices."
I guess my final question is: Is Partition-Based sync staying around long-term? Because it's way, way better.

We are planning on phasing out partition-based sync on the server, and sometime in the not too distant future orgs which don't already have a PBS app won't be able to create new ones.
With recent versions of Realm you can connect to a flexible sync server app using the partition-based API and we automatically generate the required subscriptions. This was intended as a migrational thing, but I suppose it isn't limited to that.
I'm in full agreement that the APIs we have now aren't great for app-first development where you don't already have a backend with way more data than is reasonable to ship to each client. Some of the relevant people are currently out on holiday, but when they return we can hopefully talk about ways to improve this.

Untested, but this should automatically subscribe to all objects for all types in your schema:
func subscribeToAll(_ subscriptions: RLMSyncSubscriptionSet) {
    let schema = RLMSchema.shared().objectSchema
    // This check only works if you never remove a type from your schema
    if subscriptions.count == schema.count {
        return
    }

    for objectSchema in schema {
        let name = objectSchema.className
        if !subscriptions.subscription(withName: name) {
            subscriptions.addSubscription(withClassName: name, subscriptionName: name,
                                          predicate: NSPredicate(value: true))
        }
    }
}

let config = user.flexibleSyncConfiguration(initialSubscriptions: ObjectiveCSupport.convert(object: subscribeToAll), rerunOnOpen: true)

It turns out our Swift API doesn't really support this sort of dynamic usage due to only exposing typed wrappers of the functions, so this has to drop down to the obj-c API.

I'll certainly second providing a simpler - SYNC ALL - config option.  Our app also needs to just sync all objects.
@tgoyne - now that I have solved my data loading problem with js I am updating the RealmSwift client app - what is the correct query syntax for syncing all objects ?
subs.append( QuerySubscription<AppPermission> { ????  Query to sync ALL OBJECTS ???
Thanks. It seems the sync download progress might have been fixed now so will look to migrate to flexible sync over the next 6 months.


Please help me assign a click event on each html element.

Hey members, I am getting an issue while applying the onclick event on each html element in the react. Please check the code that I am using and give me better solution.
Here is the code:
<DeepChat
style={{
width: "100%",
flex: 1,
height: "100%",
}}
ref={chatElementRef}
key={params.chatId + "chat" + params.threadId}
demo={true}
initialMessages={[
{
text: "Hello",
type: "text",
role: "bot",
timestamp: new Date(),
html: "Hello",
},
...history.map((message) => ({
text: message.text,
type: "text",
role: message.role,
timestamp: message.timestamp,
html:
message.role === "user" &&
`
<button
class="view_logs"
style="color: #0377FB; display: flex; background-color: #fff; border: none; cursor: pointer;"
               > View Logs </button>
                                      `,
          })),
        ]}
        request={{
          handler: async (query, signal) => {
            console.log(query, "from request handler");
            try {
              const response = await handleSubmitQuery(query);
              const botData = response[1];
              return signal.onResponse({
                text: botData.content,
              });
            } catch (err) {
              return signal.onResponse({
                error: err.message,
              });
            }
          },
        }}
        submitUserMessage={(message) => {
          console.log(message);
        }}
        messageStyles={{
          html: { shared: { bubble: { backgroundColor: "unset" } } },
        }}
      />

I am using this as a component in the react and it is working fine but the button in the html element should trigger an onclick. Could you please help me with this problem?

Hi @Ankur-2002.
To apply events to your custom html elements, you will have to use the htmlClassUtilities property. As an example, referring to your project - you can use the following to help get you started:
htmlClassUtilities={{
  'view_logs': {
    events: {
      click: () => {
        console.log('view logs')
      },
    }
  }
}}

I will be closing this issue as there has been no further activity.


Feature addition: Verilog code in Python strings along with verilog files.

The current API is such that it takes in a (list of) file(s) as input. However,
when embedding Pyverilog in another python libraries/code, it would be helpful
to pass python strings directly to the API rather than files.
This feature works along with the current implementation in such a way that
the user can either continue passing files as inputs as in the existing API
or pass Verilog code in python strings to the API.
Addresses #63

I'm sorry for the very long delay in my response. Thank you very much for your interesting and useful request. I am very pleased to pull your request!

Thank you very much. The library has really helped a lot!


Support loading and fine-tuning TensorFlow models

We should create a guide on how to load TensorFlow models and integrate them into a Flax training workflow for fine-tuning. For now we should avoid creating additional tooling.
References:

call_tf
deepmind/tf2jax

One follow up: it would be nice to also support something like Haiku's to_module (https://dm-haiku.readthedocs.io/en/latest/api.html?highlight=to_module#to-module), which I've found useful for incorporating TensorFlow modules (e.g., tf2jax + hk.to_module).


Possible Memory Leak When Invoking DSC Configuration (7.1-preview1 on ubuntu)

I have started using Powershell 7.1.0-preview.1 to compile DSC configurations into mofs on Linux and have noticed a potential memory leak when invoking a configuration block.
Steps to reproduce
# Uses PsDscResources which isn't builtin because it works on Linux/PsCore
Configuration TestConf {
    Import-DscResource -ModuleName 'PSDscResources'

    Node 'localhost' {
        Environment 'test' {
            Name = 'foo'
            Ensure = 'absent'
        }
    }
}

while ($true) {
    & TestConf
    Start-Sleep -Milliseconds 200
    $ws = (Get-Process -Id $pid).WS
    Write-Warning "$($ws / 1MB)MB in use"
}

Expected behavior
MB in use should stay roughly the same indefinitely.

Actual behavior
MB rises slowly but steadily until OOM

Environment data
$PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.1.0-preview.1
PSEdition                      Core
GitCommitId                    7.1.0-preview.1
OS                             Linux 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

I have tried a few things to stop this such as calling Remove-Item function:/TestConf and [System.GC]::Collect() but these do not appear to have any effect. I'm happy to try any other steps or reproduce on another version/OS if it helps identify the issue.

It is probably also worth mentioning that when running the above I get warning messages that Embeded resource are not support on PSCode.... I don't know if that is linked in some way to the issue but it feels like they might be left over from a previous release where DSC wasn't supported on Linux. As far as i can tell the mof does get created correctly.

Hey @mrhockeymonkey,
Can you try adding the following between runs to see if that's still an issue:
[Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache]::ClearCache()
[Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache]::Initialize()

Hey @gaelcolas,
Thanks for the suggestion. Just gave it a go and it does not appear to solve the issue. The memory usage still creeps up.


Add missing windows.h header

To avoid errors about unknown type name DWORD/PBYTE/LONG/etc

Microsoft DWORD/PBYTE/LONG/etc

Access


improve cmake support for msvc

Why
Fix this error
$ cmake -DASYNC_SIMPLE_BUILD_MODULES=ON -G Ninja ..

-- Skipping benchmarks
-- Configuring done (16.1s)
CMake Error in modules/CMakeLists.txt:
  The "async_simple_modules" target has C++ module sources but its
  experimental support has not been requested

-- Generating done (0.1s)
CMake Generate step failed.  Build files cannot be regenerated correctly.

https://github.com/alibaba/async_simple/pull/320#issuecomment-1579825152
What is changing
Example

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[TASK] Prepare the 8.7.0 release

Sigh. It seems we still need to fix some more places after all for PHP 8.4 compatibility for Emogrifier. (I wonder why the tests  for PHP-CSS-Parser do not catch this.)
https://github.com/MyIntervals/emogrifier/pull/1304

Sigh. It seems we still need to fix some more places after all for PHP 8.4 compatibility for Emogrifier.

Possibly needs something for #677.

(I wonder why the tests for PHP-CSS-Parser do not catch this.) MyIntervals/emogrifier#1304

That is indeed strange, since we are running unit tests with PHP 8.4, which will surely call CSSList\Document::render().  Are deprecation warnings disabled in the PHPUnit config?

Are deprecation warnings disabled in the PHPUnit config?

Indeed: https://github.com/MyIntervals/PHP-CSS-Parser/pull/744

This should be ready to go now. Emogrifier also is green with the latest v8.x branch: https://github.com/MyIntervals/emogrifier/pull/1304 (ignore the "lowest dependency version" failure)

Sigh. And another nullable-by-default parameter that isn't covered by the the parser tests yet: https://github.com/MyIntervals/emogrifier/actions/runs/11538536814/job/32117129042?pr=1304

Now the Emogrifier CI build is green with the latest changes in the v8.x branch of PHP-CSS-Parser: https://github.com/MyIntervals/emogrifier/pull/1304
I think 8.7.0 is ready to get released.

I have just released version 8.7.0: https://github.com/MyIntervals/PHP-CSS-Parser/releases/tag/v8.7.0


I refactored the project

I refactored and modernized the whole codebase. The reason is that I plan on actively using this on my company and I would like to help maintain and extend the module in the future (for example, with the addition of aws EC2 and IAM auth support).
The main changes are:

Migrated to standardjs for simplicity (less boilerplate, more standard).
~Using code up-to-date with the latest LTS Node version (v8), that allows ES6 and ES7 features like the spread operator and async / await syntax.~ Removed for node v6 compatibility.
Organized, commented and cleaned up a lot of little things.
The most notable design change is the use of an ES6 class that is not exported directly, but used to keep state. A method of the class called _createClient returns the 'client' object where all the features, methods and properties are explicitly exposed. The function that the module exports creates an instance of this class, creates a client through _createClient and returns it.

About breaking API changes, there are none except that the methods that are supposed to be private (like the legacy generateFunction, now called _generateFeature, and similar) are now actually private.
All the tests are passing and coverage is the same, only one non-critical test is outdated because of design changes, pending a rewrite that shouldn't block progress.
All package.json scripts tested and I even played around a little bit with the examples.
Some pending stuff:

[x] Refactor features.js a bit
[x] Deeper refactor (most formal logic is untouched)
[x] Make sure everything is documented and clear
Other stuff that will be added down in the comments

Let me know what you think :)

@DaniGuardiola I will do a code review tomorrow 😉 .

@kr1sp1n awesome! The PR is not as prio for my company as it was, so maybe we can add more features and functionality and further improve code, tests and documentation before releasing a major version. We even could (and probably should) create a few release-candidates to let the users test it and give some feedback.
Let me know what you think :)
And btw, we need to talk about automated/inferred logins. I have an experiment going on that we are actually running in production in our company, the aws-auth branch: https://github.com/DaniGuardiola/node-vault/tree/aws-auth
Take a look specifically at this commit: https://github.com/DaniGuardiola/node-vault/commit/6bcfbacc293a27207db264eb74cf07f4fa15a7bb
I think this is the way to go in the future. The end goal is to remove as much boilerplate as possible for authentication. As we are moving to a container orchestrator, a similar approach will be added for orchestrator-based login (using approle I think), with the corresponding aliases and etc.
These and other automated login methods could be included in 1.0

@DaniGuardiola Your proposals sound like we both should do some technical designs together 😉
I would love to work on the inferred logins as a new feature so I created a Version 1.0.0 Project:
https://github.com/kr1sp1n/node-vault/projects/2
Let's use this board to keep track of all the changes (code, tests etc.) we want to see in this major release. WDYT?

@kr1sp1n sounds good! Could you add me to the project as contributor so I can edit and move stuff around? :)

@DaniGuardiola Added you as a collaborator: https://github.com/kr1sp1n/node-vault/invitations

@kr1sp1n what needs to happen for this PR to move forward? I’m using this branch in production and I’d love to load it from NPM.

@kr1sp1n +1 approving changes so this can be merged in

Sorry guys, I haven't had time and I guess @kr1sp1n hasn't either. I'll try
to block some time at some point to make it happen, there's actually not
much work left and it could be done in an hour (there's some minor
documentation missing, and the aws inferred auth thing and that's about
it). I hope I can do it soon rather than late, but can't promise anything.
On Mon, Jun 25, 2018 at 9:08 PM John<EMAIL_ADDRESS>wrote:

@kr1sp1n https://github.com/kr1sp1n +1 approving changes so this can be
merged in
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kr1sp1n/node-vault/pull/78#issuecomment-400061448,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIc892NgMEld7QsNnpsisNNKNuNC8F6Dks5uATUWgaJpZM4RVlZW
.

I was not trying to put any pressure, and I really appreciate the work being done here!  Just wanted to relate that I too was looking out for this to get merged in.  Again, thanks for your time maintaining this!  it's one of those low level SDKs that gets used by many maintained by few.

Hey guys, thanks for all the hard work you've put into this project thus far. 👏
Let's get this merged! 🎉

Hey @kr1sp1n what do you say we set a datetime for a call one of these weekends and we get it done in two hours by working in sync?

Hey @DaniGuardiola I think it's a great idea for getting things done but unfortunately I prepare myself for a big journey that starts in August. So no time at all 😢  Maybe you could fix and merge that stuff you wrote by yourself? Would be great!

@kr1sp1n gotcha, will see what I can do then. I still need you for the npm release!

It's possible for multiple people to have access to push versions to NPM: https://docs.npmjs.com/cli/owner

@GeoffreyBooth thanks for the hint 😜
Ok, @DaniGuardiola what's your npm user name?

@kr1sp1n https://www.npmjs.com/~daniguardiola :)

@DaniGuardiola done ✅

Does this mean a merge is close?  Any way I could help here?  about to start a project and would love to use the refactored version instead of the old one.

@jhnlsn yes, a merge is close. I need to find time.
However, the interface will remain the same so don't worry, you can use the current version and this refactored version should work as a drop-in replacement when it's out in NPM.

Is this merge still happening?

I don't think it's going to happen.

@DaniGuardiola Do you have any updates on this merge? This is a project I'd like to contribute to but I'm not sure if I should be make my changes to current project's master or this fork (which will be eventually be merged in)
Any advice would be great.

I have released a package that extends node-vault and adds AWS IAM authentication working on ECS, EC2 and lambda https://github.com/Kamahl19/node-vault-aws-iam-auth . All attempts mentioned above only work on EC2.

This thread is incredibly frustrating seeing so many affirmations that this would get merged followed by no action 😆😅

@bfaulk96 it might make sense to fork off of this project since it looks like it's been abandoned at this point.  I wouldn't mind taking that on.

@jhnlsn I don't know that I could commit to keeping it updated consistently, but I imagine I could at the very least take a look more than once every 6 months and merge simple PRs that have been neglected here 😆

@DaniGuardiola has a lot of GH contributions even as recently as May and June, so I feel like it's safe to say this project is no longer a priority

thx boys, this has been interesting to see two years later haha
hope it works out! I gave up all hope of seeing this merger, but hope to see your fork!

thx boys, this has been interesting to see two years later haha
hope it works out! I gave up all hope of seeing this merged, but excited to see your fork!

Funny enough, I just stumbled onto this PR.
My org is currently having an issue with a  different PR; A JS max call stack exceeded error that has a tiny fix submitted in February and never merged. Either way, it's very clear that this repo could use some love, even if that means it must be forked.

@kr1sp1n is the current maintainer of this project.  I'll give a final 24 hours to see if he is willing to hand over the project in the current repo.  At that point i'll go ahead and fork into an org and move forward.  I own other open source projects etc so don't mind carrying the torch for this one if need be, we can fan out to other maintainers too.

Worth noting that @kr1sp1n merged the PR I mentioned him in, so he is around 😄
Maybe we could request some more maintainers be added to help with the workload and keep this maintained a bit more?

I guess he is still around. Just saw a flurry or fixes in the past hour.

I have different priorities and not a lot of time to look into this nowadays, but if anyone wants to pick this up I'd be happy to help review it and get it merged. I just won't be able to work on it directly. Something critical, though, will be back porting all fixes and features that were added on master since this branch was last updated.

Hi @DaniGuardiola, the PR is stale for a while now and also conflicting, do you believe you'll find the time to see it through or can I close it in the meantime?

@aviadhahami got it.
I maintain my disposition to help merge this if someone's up to the task. @aviadhahami you seem to be a new maintainer around here? Would you be down to leading this effort? :)


Errors with Image transfer through dlib::bridge

Problem
I am trying to transfer up to 2MP images over dlib::bridge. I wrote a simple applications for server (the one who accepts images) and client (the one who sends images). The example works fine when i transfer relatively small images (e.g. 256x256).
However as the size of image increases the recieved image becomes only partially transfered: to bottom part if image is set to zero (see examples).
My suggestion was that the bridge object in client (sender) app dies before the transfer is completed, however, inserting a sleep after pipe::enqueue(...) call didn't help.
I am not sure if this a bug in dlib::bridge or improper use of the class.
Code
client.cpp:
#include <thread>
#include <string>
#include <iostream>
#include <chrono>

#include <dlib/bridge.h>
#include <dlib/array2d.h>
#include <dlib/pixel.h>
#include <dlib/image_io.h>
#include <dlib/image_transforms.h>

int main()
{
  // setting up pipe
  dlib::pipe<dlib::array2d<dlib::rgb_pixel> > out_image(1);
  dlib::bridge b_img(dlib::connect_to_ip_and_port("<IP_ADDRESS>", 3323), dlib::transmit(out_image));

  // generating image crops to send
  std::cout << "preparing image to send" << std::endl;
  dlib::array2d<dlib::rgb_pixel> image;
  try
  {
  dlib::load_bmp(image, "/sample_pictures/cone_scene.bmp");
  }
  catch( dlib::image_load_error & )
  {
    std::cerr << "error reading image from hard drive" << std::endl;
  }

  dlib::array2d<dlib::rgb_pixel> crop1;
  dlib::array2d<dlib::rgb_pixel> crop2;
  dlib::assign_image(crop1, image);
  dlib::assign_image(crop2, dlib::sub_image(image, dlib::rectangle(400, 800, 700, 1200)));

  // here we send images
  std::cout << "sending image of size (" << crop1.nc() << ", " << crop1.nr() << ") ..." << std::endl;
  out_image.enqueue(crop1);
  std::this_thread::sleep_for(std::chrono::milliseconds(5));
  // second one
  std::cout << "sending image of size (" << crop2.nc() << ", " << crop2.nr() << ") ..." << std::endl;
  out_image.enqueue(crop2);
  std::this_thread::sleep_for(std::chrono::milliseconds(5));
  return 0;
}

server.cpp:
#include <thread>
#include <string>
#include <chrono>

#include <dlib/bridge.h>
#include <dlib/array2d.h>
#include <dlib/image_io.h>
#include <dlib/pixel.h>

typedef dlib::array2d<dlib::rgb_pixel> img_type;

class MessageReciever : public dlib::threaded_object
{
public:
  explicit MessageReciever(dlib::pipe<img_type> &inPipe, int max_recieved_count_ = 10)
    : in(inPipe)
    , recieved_count(0)
    , max_recieved_count(max_recieved_count_)
  {}

  void thread() override
  {
    img_type value;
    while(recieved_count < max_recieved_count)
    {
      in.dequeue(value);
      ++recieved_count;
      callback(value);
    }
  }

  void callback(const img_type &val) const
  {
    std::cout << "rgb image recieved: (" << val.nc() << ", " << val.nr() << ")" << std::endl;
    std::ostringstream ss;
    ss << "recieved_" << std::setfill('0') << std::setw(2) << recieved_count << ".bmp";
    std::string filename = ss.str();
    std::cout << "saving to " << filename << std::endl;
    dlib::save_bmp(val, filename);
  }
private:
  dlib::pipe<img_type> ∈
  int recieved_count;
  const int max_recieved_count;
};

int main()
{
  dlib::pipe<img_type> in_img(1);
  dlib::bridge b_img(dlib::listen_on_port(3323), dlib::receive(in_img));

  MessageReciever img_reciever(in_img);
  img_reciever.start();
  img_reciever.wait();
  return 0;
}

CMakeLists.txt
project (pipe_test)

# some line to include dlib into cmake project
include(../dlib/cmake)

add_executable(pipe_test_client client.cpp)
target_link_libraries(pipe_test_client  dlib)

add_executable(pipe_test_server server.cpp)
target_link_libraries(pipe_test_server  dlib)

test data
Original image that is being sent (i had to convert them to png manually to be able to upload to github):

results:
I start server.exe, then run clinet.exe for several times.  i get following output:
rgb image recieved: (2048, 1088)
saving to recieved_01.bmp
rgb image recieved: (2048, 1088)
saving to recieved_02.bmp
rgb image recieved: (2048, 1088)
saving to recieved_03.bmp
rgb image recieved: (2048, 1088)
saving to recieved_04.bmp
rgb image recieved: (2048, 1088)
saving to recieved_05.bmp
rgb image recieved: (2048, 1088)
saving to recieved_06.bmp
rgb image recieved: (301, 288)
saving to recieved_07.bmp
rgb image recieved: (2048, 1088)
saving to recieved_08.bmp
rgb image recieved: (301, 288)
saving to recieved_09.bmp
rgb image recieved: (2048, 1088)
saving to recieved_10.bmp

This shows that sometimes the smaller image is lost. Here are the artifacts on the images that were recieved:

System and versions
I tried following setups:

PC with Intel Core i7-4790, 16GB RAM, Windows 7 Service Pack 1.
Visual Studio 2012 x64 + dlib 18.14 / Visual Studio 2015 x64 + dlib 19.4
for server and client apps running through localhost

Laptop with Intel Core i7, 8GB RAM, Fedora Linux Release 20 x64
gcc 4.8.3 + dlib 18.14
for client app sending data to windows pc via ethernet connection.

In case with real ethernet connection situation reproduced on even smaller images.
what did i come up with so far

Adding this_thread::sleep doesn't help.
I tried running visual studio debug for both server and client, but couldn't find anything - i went up to calls to WaitForSingleObject. I aslo noticed that when I am at breakpoints in both server and client, all messages miss.

You can't just terminate the program.  If the bridge is still sending data it's going to kill the transmission part way though.  You have to wait for it to be done, which requires some kind of acknowledgement from the other side that it's got the data.

@davisking ok, i'll try improving the example and see if it works.
I thought sleeping for 5 msec would be enough for connection via localhost.
Well at least now you have sources for "sending images through bridge" example .=)

There is no way to manage transmission end.    It is not reasonable to assume that the bridge destructor would have some kind of data acknowledgement since that might hang forever waiting for acknowledgement from the other side that it got the data.  It should be obvious that the bridge doesn't do this since the bridge destructor doesn't hang.
It should also be noted that even if the bridge gives all it's data to the OS and all the data looks "sent" that doesn't mean the other host eventually gets the data.  If your application terminates the OS is perfectly allowed to drop those outgoing data buffers without sending them, which is often what it will do if you terminate your process before those OS networking buffers are done sending.  There are also a lot of other ways the data might not eventually arrive at the other side.  So you have to wait for some kind of acknowledgement back from the other end if you want to be sure it got the data.


Allow selecting your model on the main page

Is your feature request related to a problem? Please describe.
You have to go into settings to change your model but sometimes you want to change it frequently because a faster less intelligent model can do most of the searches but occasionally you need to switch it to a smarter model.
Describe the solution you'd like
A control to select your model on the main search page that allows you to start typing a model name and autocompletes for you.
Describe alternatives you've considered
I've used online competitors such as perplexity but I find perplexica better.

Maybe drop this in https://github.com/ItzCrazyKns/Perplexica/issues/247, I might consider moving things here and there.


Adding search form to 404 page

Adding search form to 404 page is a good ideas because people search them if article is not founded by website

yes, but your codes are wrong. and can you provide some 'css styles'?

sorry for wrong codes and added correct code


GUI: Fix french translation file for 0.17.2

Manually fixed 3 format specifiers which can't be fixed on transifex

If you'll pull translations from transifex again, please merge this PR after that.


examples/inspiration section on the documentation website displays a blank page on scroll

Bug is related to

[ ] embla-carousel (core package)
[ ] embla-carousel-react
[ ] embla-carousel-autoplay
[ ] embla-carousel-auto-height
[ ] embla-carousel-class-names
[x] embla-carousel-docs (documentation website)

Embla Carousel version

v1.28.1 ? I don't know which one you use on your website :)

Describe the bug

As soon as I scroll the examples/inspiration section, I get a blank page and a console error

CodeSandbox

https://www.embla-carousel.com/examples/inspiration/

Steps to reproduce

Go to https://www.embla-carousel.com/examples/inspiration/
Scroll

Expected behavior

It should not display a blank page :D

Additional context

Broswer: Chrome, Version 100.0.4896.127 (Official Build) (x86_64)
Device: Macbook Pro MacOS Big Sur v11.6.5

Thank you for your bug report @CamilleHbp! I’ll look into this as soon as possible.
Cheers,
David

Hello again @CamilleHbp,
Thank you very much for noticing this so quickly. I'm working on v7:

#298

...so the Gatsby cache that was preserved from the v7 branch made its way into the master branch.
Cheers,
David


Add cypress test instructions to the README

Also removed an outdated reference to PureCSS, which we ditched ages ago.

Sure, good point.


Refactor the per-cluster CT maps manager

Let's simplify the per-cluster CT maps management logic, removing the dependency on the global variable (which does not play well with the hive framework) and splitting the creation/removal tasks from the retrieval of inner maps.
While being at it, let's also slightly improve the performance around these operations by avoiding to attempt opening the inner maps when not strictly necessary, as well as keeping the list of known Cluster IDs.
Additionally, let's also move the fake implementation for testing purposes to a separate package, for better separation.
Finally, let's validate the ClusterID in the bpf ct list cluster <ID> CLI command, to properly return an error in case it is invalid.
Related: https://github.com/cilium/cilium/pull/27430
Depends on: https://github.com/cilium/cilium/pull/27356

/test

@YutaroHayakawa PTAL

The last push introduced the additional clarification comments from the discussion above.

/test

/test

@tklauser ping


[Bug]: can't use with git-crypt?

Describe the bug
I have installed git-crypt and using it with my repo. it works perfect except for when doing so through obsidian-git, which gives the error below.
I can't figure this out, because it seems that somehow obsidian-git cannot find the executable for git-crypt which is in /usr/bin/git-crypt
Relevant errors (if available)
Uncaught (in promise) Error: "git-crypt" clean: line 1: git-crypt: command not found
error: external filter '"git-crypt" clean' failed 127
error: external filter '"git-crypt" clean' failed
fatal: .obsidian/plugins/obsidian-git/data.json: clean filter 'git-crypt' failed

    at Object.action (plugin:obsidian-git:23985:25)
    at PluginStore.exec (plugin:obsidian-git:24010:25)
    at eval (plugin:obsidian-git:21571:43)
    at new Promise (<anonymous>)
    at GitExecutorChain.handleTaskData (plugin:obsidian-git:21569:16)
    at GitExecutorChain.eval (plugin:obsidian-git:21553:44)
    at Generator.next (<anonymous>)
    at fulfilled (plugin:obsidian-git:20402:24)

Steps to reproduce
set up git-crypt successfully using the standard way. All working as expected using git pull or push with user commands.
e.g. in .git/config
[filter "git-crypt"]
smudge = "git-crypt" smudge
clean = "git-crypt" clean
required = true
[diff "git-crypt"]
textconv = "git-crypt" diff
Then, toggle on obsidian-git from within Obsidian plugins. It fails to find git-crypt. Even if I update the config with full path as /usr/bin/git-crypt
Expected Behavior
obsidian-git upon toggling should just be able to find git-crypt. or tell us how we can make use the system installed git and git-crypt?
Any help would be appreciated.
Addition context
No response
Operating system
Linux
Plugin version
2.16.0

Have you Obsidian installed via flatpak?

@Vinzent03 nice one! yes that was indeed the issue. I used AppImage and it is fine. Except now everytime I save and obsidian-git auto backsup , it somehow pushes everything in plain text!
If I do git commit and then push on the terminal directly however, then it is encrypted. So it seems like for some reason obsidian-git auto backup will push to repo without doing any of the crypt process.
Yet, there are no errors in the console.

I'm not familiar with git-crypt. For commit etc you use the standard git binary, but with the special config in .git/config ? The plugin should respect the config file.

@Vinzent03 thanks for your help. The setup works everywhere with git crypt.the only place it fails is on Android. Guess it's just the limitation for now

Yeah it not gonna work on mobile, because there is no real git nor git-crypt installation on mobile.

@Vinzent03 FYI you can get git crypt on mobile. In fact. I did it through termux and.got it working.it just doesn't work if through obsidian git

Yep, it's what I was searching about. Tnx u

@Vinzent03 FYI you can get git crypt on mobile. In fact. I did it through termux and.got it working.it just doesn't work if through obsidian git

have you found any solution of how to run git-crypt through obsidian-git mobile ??


OSError: [Errno 5] Input/output error

Hi, yfukasawa,
I am running LongQC for a batch of my Pacbio unaligned BAMs.  For the small samples, it run smoothly. But for one of a little bit large BAM file, it take a very long time and finally report OSE error. My command is
longqc sampleqc -x pb-sequel -p 8 -o  HG002_90pM_read_LongQC ./m64304e_211014_201856.reads.bam
I cut the error related message as the following.   Could you help me figure out the reason?      Additionally, it take close two weeks to run this sample and get such an OSError, whether I can specific more nodes (such as -p  32)   to linearly speed up? Thank you in advance.
Wenchao
File "/opt/LongQC/longQC.py", line 63, in main
args.handler(args)
File "/opt/LongQC/longQC.py", line 829, in command_sample
tpl = env.get_template('web_summary.tpl.html')
File "/opt/conda/lib/python3.9/site-packages/jinja2/environment.py", line 997, in get_template
File "/opt/conda/lib/python3.9/site-packages/jinja2/environment.py", line 958, in _load_template
File "/opt/conda/lib/python3.9/site-packages/jinja2/loaders.py", line 125, in load
File "/opt/conda/lib/python3.9/site-packages/jinja2/loaders.py", line 201, in get_source
OSError: [Errno 5] Input/output error

BTW, I run LongQC at LSF cluster. And the version is :  LongQC 1.2.0c
Wish to hear from you.

Hi @wzhang42,
Thank you for your interests in our tool.
For a large file, this is typical for subreads.bam from Sequel II, indeed it should take much longer time and larger RAM. Because LongQC computes overlap between subsampled reads (5k by default) and all reads, the computational time is actually a bit proportional to input size.
Having said, the real cause seems to be different in your case. From a file name, I assume you applied a new file format in Sequel II (most likely Sequel IIe) so-called reads.bam.
This is a new file format introduced recently, and actually LongQC ver. 1.2.0c is not yet able to handle this format (an item on my plate). reads.bam has both CCS (HiFi) and subreads, and you would need to extract just HiFi/CCS reads from your file.
This doc might be of your help: https://ccs.how/faq/reads-bam.html
More specifically, you can follow this tutorial to get HiFi reads from reads.bam.
Once you get HiFi reads from reads.bam, please use -x pb-hifi option for the new file. then, I believe this issue will dissappear.
I hope this helps.
Yoshinori

Hi, Yoshinori,
Many thanks for your reply.  Yes, our data is reads.bam from Pacbio Sequel. I can use bam2fastq to convert the reads.bam to .fastq. For some small size file, it's ok either I use reads.bam or the converted .fastq. I felt that the main issue is the file size (file size corresponding to the read number,  the aligned .bam can be as large as 900G). I agree that it's one possible option to convert the large reads.bam to ccs.bam/hifi_ccs.bam, which will reduce the read number and file size.  But we are now interested in the LongQC metrics of the original reads.bam.    I am interested in the parameter of subsampled reads, which I believe should can significantly reduce the required RAM and running time. Could you share with us the command line to configure this parameter(option)?  I just guess that it's " -n NSAMPLE" (0-10000), default with 5000, right? If my guess is correct, then if I configure this parameter from the default 5000 to 1000, the running time will be linearly reduced to 1/5 of the original running time, Right? Additionally, whether the longQC sampleqc -p nNODE for one large .bam can be parallelized and the running time can be linearly reduced ?
Thank you again.  Much appreciated!
Wenchao

Hi Wenchao,

If my guess is correct, then if I configure this parameter from the default 5000 to 1000

Generally speaking, yes. Actual computation time relies on minimizers in your dataset, but in any case, it will be reduced.

can be parallelized and the running time

bottleneck would be minimap2 computation in the flow, and minimap2 is not designed to parallelize in multiple nodes. technically, another layer can be added, but at least it's not available in the current version.

But we are now interested in the LongQC metrics of the original reads.bam.

Thank you for your interest in our tools in this regard. This is actually a new challenge, and reasons I recommended extraction of HiFi is not just data size. Most of mapping software tools assume a single data profile, and this is common for output file from machines. but, as long as I know, reads.bam is the very first exception. -x pb-hifi and -x pb-sequel use different parameters, and either profile doesn't work well for the other profile. in reads.bam, multiple types of reads requiring different params actually co-exist to reduce file size (personal communication). Either -x pb-hifi or -x pb-sequel would not return proper statistics for a whole reads.bam.
You can also extract non-hifi reads from reads.bam, and run our tool against them using pb-sequel mode. I expect it would have some bias as non-hifi reads are results of worse sequencing in ZMWs for a HiFi run. Stats would look worse than the real condition. I haven't tested this yet, but for now this is another technical reason I recommended to extract HiFi.
I hope this helps.
Yoshinori


Were JS file access functions left out?

Just tried to use fetch() and FileReader, but they both error. Were they left out in favor of using the godot file functions?

Nevermind, I'll post a better question


win-19 low on disk space for Temp folder.

Type of Request
bug
Infrastructure Environment
LUCI, Github, Cocoon scheduler, Autosubmit, etc...
What is happening?
win-19 low on disk space for Temp folder.

Steps to reproduce
Step 1:
Step 2:
..
Step n:
Expected results
I expect to see X when Y is finished.

Removing old files and folders under Temp directory to free up disk space.


How to inference after finetuning ?

Thanks for sharing the training code. I've finished a 3-epoch finetuing.
However, I don't find the inference code.
Would you please give some advice on it? or sharing the infercence code ?
Thanks again!

May I ask, what version of transformers do you use?

a third-party fork as describe in  https://github.com/tatsu-lab/stanford_alpaca#:~:text=Given Hugging Face hasn't officially supported the LLaMA models%2C we fine-tuned LLaMA with Hugging Face's transformers library by installing it from a particular fork (i.e. this PR to be merged). The hash of the specific commit we installed was 68d640f7c368bcaaaecfc678f11908ebbd3d6176.

@kriskrisliu Are you able to generate text normally? I got something like "bashselectionièmeRemove₃ahractory Normdateiering declaringйской autom ін annual (+ групў "

@kriskrisliu Thanks for the reply! I was fine-tuning using the dataset they provided.
And this are the output I got:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction:
{How are you doing}
{{\ cincoapk Sorry Past estimationerca foreverтелейcred suo surely Kriegsœ Toulclihmгуnamespaceifecycle generateRES Insteadorter fosse lect exclus Bowl IX Нью proves monarchBU Liguedefn pedigclean Pok Japon Okayун primit "- educational sorte movie folders communicate Quando italiana Ej Турнва Étatsonn3a....
When I was training the model, it throws an error after finishing the last epoch:
packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2224: UserWarning: Failed to clone() tensor with name _fsdp_wrapped_module._fpw_module.model.layers.29.mlp.up_proj.weight. This may mean that this state_dict entry could point to invalid memory regions after returning from state_dict() call if this parameter is managed by FSDP. Please check clone implementation of _fsdp_wrapped_module._fpw_module.model.layers.29.mlp.up_proj.weight. Error: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 0; 39.59 GiB total capacity; 36.73 GiB already allocated; 148.19 MiB free; 37.48 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
But it looks like the model is still saved after this error.

@kriskrisliu
And I was using this command:
full_completion = model.generate(inputs=tokenized_text["input_ids"].to("cuda"),
attention_mask=tokenized_text["attention_mask"].to("cuda"),
temperature=0.7,
top_p=0.9,
do_sample=True,
num_beams=1,
max_new_tokens=600,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id)

I didn't meet these errors.
Results from my finetuned model are like:
Instruction: List all Canadian provinces in alphabetical order.
Response: Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Nova Scotia, Ontario, Prince Edward Island, Quebec, Saskatchewan, Yukon.</s>

Instruction: Write a Python program that prints the first 10 Fibonacci numbers.
Response: def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)</s>

compare with the non-finetuned: (you can see that the model sometimes repeats the question and give low-quality response)
Instruction: List all Canadian provinces in alphabetical order.
Response: Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince Edward Island, Quebec, Saskatchewan, Yukon
### Instruction:
List all Canadian provinces in order of population.
Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince Edward Island, Quebec, Saskatchewan, Yukon
### Instruction:
List all Canadian provinces in order of area.
Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince Edward Island, Quebec, Saskatchewan, Yukon
### Instruction:
List all Canadian provinces in order of population and area.
Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince

Instruction: Write a Python program that prints the first 10 Fibonacci numbers.
Response: The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34.
The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5,

@kriskrisliu Thanks for sharing the result. Your results look very good!
I was using the following code to perform inference. Is it similar to yours?
Load model
print("Loading model...")
model = AutoModelForCausalLM.from_pretrained(directory)
tokenizer = AutoTokenizer.from_pretrained(directory)
Use FP16
model = model.half()
Move to GPU
print("Moving model to gpu...")
model = model.to("cuda")
tokenized_text = tokenizer("Instruction: List all Canadian provinces in alphabetical order.", return_tensors="pt", padding="longest", max_length=tokenizer.model_max_length, truncation=True)
Generate result
full_completion = model.generate(inputs=tokenized_text["input_ids"].to("cuda"),
attention_mask=tokenized_text["attention_mask"].to("cuda"),
temperature=0.7,
top_p=0.9,
do_sample=True,
num_beams=1,
max_new_tokens=600,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id)
decoded_text = tokenizer.decode(full_completion[0])

@kriskrisliu Are you using the main branch of HuggingFace Transformers for fine-tuning (Llama)?

@kriskrisliu It would be really nice if you can share you generation script.

@kriskrisliu Thanks for the reply! I was fine-tuning using the dataset they provided.
And this are the output I got: Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction:
{How are you doing}
{{\ cincoapk Sorry Past estimationerca foreverтелейcred suo surely Kriegsœ Toulclihmгуnamespaceifecycle generateRES Insteadorter fosse lect exclus Bowl IX Нью proves monarchBU Liguedefn pedigclean Pok Japon Okayун primit "- educational sorte movie folders communicate Quando italiana Ej Турнва Étatsonn3a....
When I was training the model, it throws an error after finishing the last epoch:
packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2224: UserWarning: Failed to clone() tensor with name _fsdp_wrapped_module._fpw_module.model.layers.29.mlp.up_proj.weight. This may mean that this state_dict entry could point to invalid memory regions after returning from state_dict() call if this parameter is managed by FSDP. Please check clone implementation of _fsdp_wrapped_module._fpw_module.model.layers.29.mlp.up_proj.weight. Error: CUDA out of memory. Tried to allocate 172.00 MiB (GPU 0; 39.59 GiB total capacity; 36.73 GiB already allocated; 148.19 MiB free; 37.48 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
But it looks like the model is still saved after this error.

Load tokenizer like train.py
tokenizer = transformers.AutoTokenizer.from_pretrained(
    origin_tokenizer,
    padding_side="right",
    use_fast=False,
)
if tokenizer.pad_token is None:
    smart_tokenizer_and_embedding_resize(
        special_tokens_dict=dict(pad_token=DEFAULT_PAD_TOKEN),
        tokenizer=tokenizer,
        model=model,
    )
tokenizer.add_special_tokens(
    {
        "eos_token": DEFAULT_EOS_TOKEN,
        "bos_token": DEFAULT_BOS_TOKEN,
        "unk_token": DEFAULT_UNK_TOKEN,
    }
)

I didn't meet these errors. Results from my finetuned model are like:
Instruction: List all Canadian provinces in alphabetical order.
Response: Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Nova Scotia, Ontario, Prince Edward Island, Quebec, Saskatchewan, Yukon.</s>

Instruction: Write a Python program that prints the first 10 Fibonacci numbers.
Response: def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)</s>

compare with the non-finetuned: (you can see that the model sometimes repeats the question and give low-quality response)
Instruction: List all Canadian provinces in alphabetical order.
Response: Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince Edward Island, Quebec, Saskatchewan, Yukon
### Instruction:
List all Canadian provinces in order of population.
Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince Edward Island, Quebec, Saskatchewan, Yukon
### Instruction:
List all Canadian provinces in order of area.
Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince Edward Island, Quebec, Saskatchewan, Yukon
### Instruction:
List all Canadian provinces in order of population and area.
Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince

Instruction: Write a Python program that prints the first 10 Fibonacci numbers.
Response: The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34.
The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5,

@kriskrisliu I think many folks here meet the same issue for inference. It would be great if you could help share the inference code. Thanks!

Could you share the ckpt you trained? Or has anyone find an existing ckpt for generation?

@kriskrisliu  could you share the code of inference？

@MrRace @XinliYu https://github.com/tatsu-lab/stanford_alpaca/pull/199 here's PR for inference code

@wade3han
Thanks a lot for your work.I meet an error like below when do inference:
    transition_scores = model.compute_transition_scores(
  File "/opt/python3.10.11/lib/python3.10/site-packages/transformers/generation/utils.py", line 1051, in compute_transition_scores
    indices = sequences[:, cut_idx:] + beam_sequence_indices
RuntimeError: The size of tensor a (114) must match the size of tensor b (259) at non-singleton dimension 1

How to fix it?

@MrRace @XinliYu #199 here's PR for inference code

@wade3han Does it support multi-turn dialogue？ Thanks a lot!

I have the same problem, but I don't know why he showed up. This problem is solved when I comment out line 116-118.

@wade3han Thanks a lot for your work.I meet an error like below when do inference:
    transition_scores = model.compute_transition_scores(
  File "/opt/python3.10.11/lib/python3.10/site-packages/transformers/generation/utils.py", line 1051, in compute_transition_scores
    indices = sequences[:, cut_idx:] + beam_sequence_indices
RuntimeError: The size of tensor a (114) must match the size of tensor b (259) at non-singleton dimension 1

I just use
instructions= [
        "模仿鲁迅的风格, 吐槽一下最近食堂饭菜涨价",
    ]

How to fix it?

I have the same problem, but I don't know why he showed up. This problem is solved when I comment out line 3.

I encountered same issue when I used different instructions.
Actually the lines _alpaca/blob/64100b4139545818b578cde7410b0ea66a62de9e/inference.py#L93-L103  are not necessary for inference (just for debugging), you can just comment out them.


feat(Object): add batch error.results

AV.Object#fetchAll，AV.Object#saveAll， AV.Object#destroyAll 遇到失败时，抛出的 error 对象增加 results 属性，类型为 Array<AV.Object|Error>
close #96
https://leanticket.cn/tickets/14146

Codecov Report

Merging #516 into master will decrease coverage by 0.09%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master     #516     +/-   ##
=========================================
- Coverage   67.11%   67.02%   -0.1%
=========================================
  Files          36       36
  Lines        2959     2972     +13
=========================================
+ Hits         1986     1992      +6
- Misses        973      980      +7

Impacted Files
Coverage Δ

src/object.js
82.44% <100%> (-0.47%)
:arrow_down:

src/op.js
56.57% <0%> (-1.32%)
:arrow_down:

src/acl.js
70.52% <0%> (+1.05%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ee8e68...2c39aa3. Read the comment docs.


Fix date parsing

Fixed:
Dates parsing with formats "dd/MM", "dd/MM/yy", "MM/yy"
Dates converting from number of days
Incorrect modification of the localization date pattern

Nice, thanks!


Update test coverage

Coverage increased (+2.3%) to 96.992% when pulling 417f8621608d785849deb02d5548a4232f0514a1 on update-test-coverage into 3ee84cee1021499179f0858354722b5507ad607a on master.

Coverage increased (+2.3%) to 96.992% when pulling 899ad086b48dce4b7fef5162948cc2a0bf8f4ae0 on update-test-coverage into 3ee84cee1021499179f0858354722b5507ad607a on master.


Results Presentation for v0.6

While this isn't the results groups, let's talk about the results table for v0.6.

SWG Notes:
Previous v0.5 results display as a table:
https://mlperf.org/results/
Changes from v0.5:

Display TTA as time in minutes (2 decimals) instead of speedup over reference -- do not display speedup.
Remove reference row
https://mlperf.org/results/ will be titled v0.6 (with month & year)
hyperlink on https://mlperf.org/results/ will point to https://mlperf.org/results_v0.5  (with month & year) which holds the previous results
Leave column for "recommendation" benchmark as a place holder with foot note explaining why it is empty/not submitted.
Describe changes and give direction on how to compare v0.5 to v0.6.
Consider: expanding the table such as host processor count, accelerator count.
Consider: submitter determined notes field.

AI - A proposal for the results format will be emailed to submitters and results mailing list next week. Blank to submitters & results WG, filled sheet to commercial submitters.

SWG Notes:

We may temporarily remove power and cloud scale -- this would not set a precedent for future submissions; this is just acknowledging that we don't have things to say about it for v0.6.
We may also report virtual CPUs for cloud submissions.

Results have already been presented for v0.6.


Human heart snRNA-seq, snATAC-seq and Visium data

Project short name: HumanMyocardialInfarctionSpatialMap
Ingest project UUID: e9f36305-d857-44a3-93f0-df4e6007dc97
Kuppe C, Ramirez Flores RO, Li Z, et al. Spatial multi-omic map of human myocardial infarction. bioRxiv; 2020. DOI: 10.1101/2<IP_ADDRESS>1686

Primary Wrangler:
Ray
Secondary Wrangler:

Associated files:
Google Drive:
Key Events

[x] initial meeting with Rafael Kramann and colleagues -- minutes
[x] generate custom metadata spreadsheet
[x] Assign the Primary Wrangler
[x] Send custom metadata template spreadsheet, how-to guide and directions for submitting data files
[ ] Receive spreadsheet (to a sufficient standard to submit dataset)
[ ] Receive data (to a sufficient standard to submit dataset)
[ ] sumbit matrix files to ArrayExpress
[ ] Curate metadata spreadsheet with ontologies
[ ] Upload sheet to validate metadata
[ ] Run graph validator & Check linking
[ ] Validate data files
[ ] Ask the Secondary Wrangler for an end-to-end review of the project. Ask the Expertise Wrangler to review specific tabs if needed
[ ] Get final approval of submission from the contributor including release date, metadata spreadsheet and data files
[ ] Submit to production and inform ingest-devs the project is ready for archiving
[ ] Email UX to inform that they're ready to send survey

Please track the below as well as the key events:

Track date first spreadsheet received and final spreadsheet sent by editing ticket to include
date next to event.
Track spreadsheet iterations by placing asterisks next to receive spreadsheet event.
Track any metadata issues/tickets made for dataset with a bulleted list of links under
received spreadsheet event. Links should be to the ticket in the metadata repo.

Some of the data is sensitive and may require controlled access.
Alternative approaches to support the contributors in sharing their data: https://docs.google.com/document/d/189A0zdttGDTc6usimWDKcvapeiFyaCTmrvjHi7d908s/

Today I sent a follow up email to Rafael Kramann..

They have decided to use a Zenodo entry to share processed data to reviewers during the submission process. They are planning to continue with the HCA submission during the paper revision process.

Stalled until contributors are ready to progress with their submission after their paper is accepted.

@rays22 to follow up on this

I have found a pre-print that I think is related to this project:
Kuppe C, Ramirez Flores RO, Li Z, et al. Spatial multi-omic map of human myocardial infarction. bioRxiv; 2020. DOI: 10.1101/2<IP_ADDRESS>1686
I have sent a follow-up email to the contributors to request cell type annotations and cell-gene expression matrix files to share via the HCA DCP.

I have not received any cell annotation or gene expression matrix files from the contributors.
The wrangling of this project is stalled, because there is not any public data available yet for this project.

Pre-curated spreadsheet, sent instructions to upload matrix and asked for review of the pre-filled template

Duplicated in #661


[WrappedPunks] Mainnet safe app not working

Description
WrappedPunks safe app is using a "eth_subscribe" RPC method that we do not support and the app fails to be opened as a consequence.
Steps to reproduce
Go to mainnet and open the safe app (it is only available for this chain).
Expected result
App opening as expected.
Obtained result

App is loading now, closing this ticket 😉


User messages

Solves inconsistencies and ambiguities in messages displayed to the user.
Close https://github.com/psyinfra/onyo/issues/423
Close https://github.com/psyinfra/onyo/issues/430
Close https://github.com/psyinfra/onyo/issues/442

Thanks for the suggestion, I updated the PR accordingly 👍

Thanks, @TobiasKadelka!


fix task reconnection issues

fixes #155163
fixes #156294
fixes #155246
fixes #156292
fixes #156284

It was working for me on macOS but isn't working for me on Windows 🤔

works on Mac but not windows for me atm 🤔
https://user-images.githubusercontent.com/29464607/182229352-61ac35cf-6eef-4061-8877-4bb4369d24bc.mov


Allow Script Tags

Sometimes the user may need to provide JS scripts (for example, adding a Google Analytics script to their site). We need a clean way to allow this.

Could I work on this? If so, could you tell me where to start?

@gringrape i guess add one more component ScriptTag to the codebase. Then we can render that component to the page. So the whole feature would be sth like this?:
script_tag = ScriptTag(url='....')

app = pc.App()
app.add_page(index, scripts=[script_tag])

What's @Alek99 idea

Is @gringrape still working on this? If no, i can help with that

Is @gringrape still working on this? If no, i can help with that.
There's a discussion on discord about how we should implement the Script tag.

Thank you! I will check on discord :)

Just for reference, I have a self-hosted umami script waiting to be implemented
'<script async defer data-website-id="3c907df2-dc16" src="https://xx.vercel.app/umami.js"></script>'
Ideally there would be something like pc.script where you could either copy&paste the whole external or define embeds?

watching this to be implemented.

Can I pick this issue up ?

@wassafshahzad Yes definitely! If you need help/guidance you can post on our Discord for faster replies.

Sure


Question : Does the new Github/Git feature contain an api or SCM provider ?

Greetings,
It would be great to provide a package that also adds support similar to the Git integration for other SCM providers such as hg and svn.
However I was not able to locate a package that contains the Git part of the feature.

This depends on what you mean by "integration". Do you mean the file change status? Or the ability to push commits and generally manage the repo?

I believe both . Mainly what they’re are considering the Git / Github new feature.  Is that structured in core or via a core package

That is handled in https://github.com/atom/github. The integration into the editor itself showing the changed lines is handled elsewhere, for svn at least there is https://atom.io/packages/svn.

Thanks for reaching out!
Because we treat our issues list as the Atom team's backlog, we close issues that are questions since they don't represent a task needing to be completed. For most questions about Atom there are a lot of options, check out:

The Atom Flight Manual
The Atom FAQ on Discuss
Discuss, the official Atom and Electron message board
Join the Atom Slack team

On Discuss and in the Atom Slack team, there are a bunch of helpful community members that should be willing to point you in the right direction.


TB-3686 Add isDismissible param to showAppBottomSheet

What 🕵 🔍

Add isDissmisable param to showAppBottomSheet.

References 📝 🔗

Jira

Codecov Report

Merging #101 (031fcb7) into main (1badb5f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #101   +/-   ##
=======================================
  Coverage   46.83%   46.83%
=======================================
  Files          58       58
  Lines        2099     2099
=======================================
  Hits          983      983
  Misses       1116     1116

Impacted Files
Coverage Δ

...src/widget/bottom_sheet/show_app_bottom_sheet.dart
0.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1badb5f...031fcb7. Read the comment docs.


test: Add vitest workspace configuration

Description 📝
Adds a Vitest workspace configuration file (vitest.workspace.ts), allowing the running and reporting of unit tests from all packages with a single command.
This change was originally suggested in the original UI package PR: https://github.com/linode/manager/pull/11057#discussion_r1792540153
Changes  🔄

Adds vitest.workspace.ts configuration file
Updates commands in package.json to run tests across all packages
Updates a single unit test that was failing when run from workspace root

How to test 🧪
In the repo root, run:
yarn test

(Should we add a M3 ticket for this to the Modularization epic?)

Added in M3-8809.


feat(goreleaser): support linux/amd64 ubuntu 24.04 builds

Address https://github.com/leptonai/gpud/issues/31.

lgtm


Added My Name

This Is A Test

Coverage remained the same at 100.0% when pulling a0d6515833a72e243c90abec6c8be1f9533b14fa on siddmegadeth:master into db7a39c97ede0f86d6033de432f2f38e9a0498ce on jirikavi:master.


Baremetal binary backend

I'm currently working on symbolication on an odd platform (homebrew on the Nintendo Switch), which doesn't use ELF or Mach (or anything standard for that matter). I was wondering if it'd be a good idea to implement a "dumb" backend that hardcodes common section names, and finds them with linker symbols. For instance, __debug_abbrev_start, __dynamic_start, etc... With weak linkage, we could get something that works pretty well, I think.
So we would create a BaremetalObject (unit?) struct that implements Object.
Would such a PR get accepted ?

Probably? As long as it doesn't complicate the rest of the crate. You mainly want the sections or sections_data_by_name methods?
I'm not sure how much it would gain you though. If you're only wanting to use the addr2line crate, I don't think there would be much boilerplate involved in passing sections to Context::from_sections instead of using this crate. We can change the object dependency in addr2line to be optional.

Hmm, yeah that's probably more reasonable.
My end-goal was to get backtrace-rs working in an embedded context, in order to use it in a custom libstd I'm working on, to report backtraces. But taking a closer look, there are many things that simply won't work (I'd also need some way to "fix" findshlibs, and I'd need to make backtrace-rs not mmap the so files. I'm probably better off using addr2line and gimli directly instead.


Textual definition update: heparin metabolic process and heparan sulfate metabolic process

This is related to the review of glycosaminoglycan metabolic process (GO:0030203) and proteoglycan metabolic process (GO:0006029) branches ticket #28977
Heparin and heparan sulfate (HS) are polymers of iduronic acid/glucuronic acid and N-acetylglucosamine (PMID: 28653068)
their synthesis is the same they only differ by:

their sulfate levels (2.3 sulfate groups per disaccharide in heparin and ∼0.8 sulfate groups per disaccharide in HS)

their localisation

HS is ubiquitously found at the surface of all animal cells
heparin is produced and stored selectively in the secretory granules of connective-tissue mast cells (mammals only?)

heparin synthesis begins on the serglycin core protein, followed by chain elongation and chain modification. In addition, the heparin chain is cleaved by heparanase (PMID: 39000302)

In summary, the genes involved in their synthesis/chain modification are likely to be the same.
Is it better to have one GO term for heparan sulfate and heparin or a separate GO term for each?
These are the current GO terms related to their metabolism belonging to glycosaminoglycan metabolic process (GO:0030203) or proteoglycan metabolic process (GO:0006029) branches:

id: GO:0030201
name: heparan sulfate proteoglycan metabolic process
def: "The chemical reactions and pathways involving any proteoglycan containing heparan sulfate, any member of a group of glycosaminoglycans that have repeat units consisting of alternating alpha-(1->4)-linked hexuronic acid and glucosamine residues, the former being a mixture of sulfated and nonsulfated D-glucuronic and L-iduronic acids, and the latter being either sulfated or acetylated on its amino group as well as sulfated on one of its hydroxyl groups." [GOC:mah, ISBN:0198506732]
synonym: "heparin proteoglycan metabolic process" RELATED []
is_a: GO:0006029 ! proteoglycan metabolic process
id: GO:0030202
name: heparin metabolic process
def: "The chemical reactions and pathways involving heparin, any member of a group of glycosaminoglycans found mainly as an intracellular component of mast cells. They are similar to heparan sulfates but are of somewhat higher average Mr (6000-20000) and contain fewer N-acetyl groups and more N-sulfate and O-sulfate groups; they may be attached in the same manner to protein, forming proteoglycans. They consist predominantly of alternating alpha-(1->4)-linked D-galactose and N-acetyl-D-glucosamine-6-sulfate residues." [GOC:mah, ISBN:0198506732]
synonym: "heparan sulfate metabolic process" RELATED []
intersection_of: GO:0008152 ! metabolic process
intersection_of: has_primary_input_or_output CHEBI:24505 ! heparins

Issues

There is no heparan sulfate metabolic process

|__mucopolysaccharide metabolic process
           |__chondroitin sulfate metabolic process
           |__dermatan sulfate metabolic process
           |__heparin metabolic process
           |__hyaluronan metabolic process
           |__keratan sulfate metabolic process

Options:

Have a unique term : heparan sulfate/ heparin metabolic process
Have 2 separate terms: heparin metabolic process and heparan sulfate metabolic process

There is no heparin proteoglycan metabolic process
|__proteoglycan metabolic process
	|__chondroitin sulfate proteoglycan metabolic process
	|__dermatan sulfate proteoglycan metabolic process
	|__heparan sulfate proteoglycan metabolic process

Options:

Update the current heparan sulfate proteoglycan metabolic process to include heparin: heparan sulfate/heparin proteoglycan metabolic process
Having 2 separate terms: heparan sulfate proteoglycan metabolic process and heparin sulfate proteoglycan metabolic process

Metabolic ressources:
In MetaCyc there is no specific pathway for heparin: superpathway of heparan sulfate biosynthesis
In Reactome,  both are under the same pathway: Heparin sulfate/heparin (HS-GAG) metabolism
In KEGG both are under the same pathway: Glycosaminoglycan biosynthesis - heparan sulfate / heparin

@rozaru
is this description of heparins in the current def of 'heparin metabolic process'  true?
They consist predominantly of alternating alpha-(1->4)-linked D-galactose and N-acetyl-D-glucosamine-6-sulfate residues
Your original post of this ticket says they "are polymers of iduronic acid/glucuronic acid and N-acetylglucosamine"

@rozaru
Since HS and heparin have different localizations, as you write in the first comment

is ubiquitously found at the surface of all animal cells
heparin is produced and stored selectively in the secretory granules of connective-tissue mast cells (mammals only?)

Maybe thy have different roles and are part of different higher-level processes, and that both terms are useful for making GO-CAMs? What do you think ?

I'm not sure what are to be merged? Isn't the proposal to RENAME
GO:0030202 heparin metabolic process -> heparan sulfate/heparin metabolic process
GO:0030201 heparan sulfate proteoglycan metabolic process -> heparan sulfate/heparin proteoglycan metabolic process
heparan sulfate/heparin metabolic process
def: "The chemical reactions and pathways involving heparan sulfate or heparin, which are members of a group of glycosaminoglycans. [REF?]"
heparan sulfate/heparin proteoglycan metabolic process
def: "The chemical reactions and pathways involving heparan sulfate or heparin proteoglycans, which are proteins covalently linked to heparan sulfate or heparin glycosaminoglycan chains. [REF?]"
Looks like child terms should also be renamed, redefined similarly.
GO:0030210 heparin biosynthetic process
GO:0030211 heparin catabolic process
GO:0015015 heparan sulfate proteoglycan biosynthetic process, enzymatic modification

There are 2 aspects for these terms: merging and renaming.
Ideally the 1st step would be to merge 'heparin metabolic process' to 'heparan sulfate proteoglycan metabolic process' to have a unique term that would be renamed 'heparan sulfate/heparin proteoglycan metabolic process' with this potential new definition:
"The chemical reactions and pathways involving any proteoglycan containing heparan sulfate, any member of a group of glycosaminoglycans that have repeat units consisting of alternating alpha-(1->4)-linked hexuronic acid and glucosamine residues, the former being a mixture of sulfated and nonsulfated D-glucuronic and L-iduronic acids, and the latter being either sulfated or acetylated on its amino group as well as sulfated on one of its hydroxyl groups. Heparin is a highly sulfated form of heparan sulfate which is specifically stored in secretory granules of connective-tissue mast cells and can be cleaved off the protein core."
However, if this goes ahead, and to be consistent, a similar step would need to be done for the following terms (see @sjm41 suggestions in ticket [28977](ticket https://github.com/geneontology/go-ontology/issues/28977) :

chondroitin sulfate metabolic process + children and chondroitin sulfate proteoglycan process + children
dermatan sulfate metabolic process + children and dermatan sulfate proteoglycan process + children
keratan sulfate metabolic process + children and keratan sulfate proteoglycan process (new term that would need to be created) + children

I am still waiting for some feedback on how to deal with the 'glycosaminoglycan metabolic process' branch as once these merges would be done it will look like this:
current:
|__glycosaminoglycan metabolic process
        |__glycosaminoglycan biosynthetic process
        |   |__chondroitin sulfate biosynthetic process
        |   |__dermatan sulfate biosynthetic process
        |   |__heparin biosynthetic process
        |   |__hyaluronan biosynthetic process
    	|   |__keratan sulfate biosynthetic process
        |   |__peptidoglycan biosynthetic process(+)
        |__glycosaminoglycan catabolic process
        |   |__chondroitin sulfate catabolic process
        |   |__dermatan sulfate catabolic process
        |   |__heparin catabolic process
        |   |__hyaluronan catabolic process
        |   |__keratan sulfate catabolic process
        |   |__peptidoglycan catabolic process
        |__mucopolysaccharide metabolic process
        |   |__chondroitin sulfate metabolic process(+)
        |   |__dermatan sulfate metabolic process(+)
        |   |__heparin metabolic process(+)
        |   |__hyaluronan metabolic process(+)
        |   |__keratan sulfate metabolic process(+)
        |__peptidoglycan metabolic process
        |__peptidoglycan biosynthetic process(+)
        |__peptidoglycan catabolic process
        |__peptidoglycan turnover

after update:
|__glycosaminoglycan metabolic process
        |__glycosaminoglycan biosynthetic process
        |   |__hyaluronan biosynthetic process
        |   |__peptidoglycan biosynthetic process(+)
        |__glycosaminoglycan catabolic process
        |   |__hyaluronan catabolic process
        |   |__peptidoglycan catabolic process
        |__mucopolysaccharide metabolic process
        |   |__hyaluronan metabolic process(+)
        |__peptidoglycan metabolic process
        |__peptidoglycan biosynthetic process(+)
        |__peptidoglycan catabolic process
        |__peptidoglycan turnover

Issue:
would this be confusing for curators/users not to find chondroitin sulfate etc under 'glycosaminoglycan metabolic process', would there be a way to link the glycosaminoglycan proteoglycan metabolic process to it?
Would discussing this on a curator/editor call help?

Also, I think we should merge:
GO:1903510 mucopolysaccharide metabolic process
and
GO:0030203 glycosaminoglycan metabolic process
See https://github.com/geneontology/go-ontology/issues/29445


Separate Graph for Upload Bandwidth

Is your feature request related to a problem? Please describe.
My ISP provides me with a much lower upload bandwidth (30 mbps) than download bandwidth (1gbps). When I view the graph on the dashboard, because the download speeds are much higher the scale of the Y axis does not allow me to perceive differences in the upload speed.
Describe the solution you'd like
I would like the option to have the upload displayed on a separate graph from the download. Or an alternative would be to have a secondary Y axis on the right hand side for upload which scales separately.
Additional context

#258