Add diagram to Cluster Architecture concept page

Description:
Updated /docs/en/concepts/architecture/_index.md and added kubernetes-cluster-architecture.svg in /static/images/docs.
Page to Update:
https://kubernetes.io/docs/concepts/architecture/

Fixes #39234

/retitle Add diagram to Cluster Architecture concept page

As seeing the changes it looks like the image already existed and what was missing is the using it. the PR resolves that but not sure what to do with the newly created architecture diagram
Also found a more verbose architectural diagram
https://github.com/kubernetes/website/blob/main/static/images/docs/architecture.svg

Hi @dipankardas011, you can visit issue #39234 for more info.

Hi @dipankardas011, you can visit issue #39234 for more info.

okay then. Good work 👍🏻

Hi @stormqueen1990 @sftim @dipesh-rawat, updated the _index.md with latest commit

/label tide/merge-method-squash

Some nits:

The spelling is incorrect: kube-api-server should be kube-apiserver;
Use either "Scheduler" or "kube-scheduler", not "scheduler".
If we use binary name for control plane components, then we should say "kube-controller-manager" rather than "Controller Manager".
The color coding is confusing. Why kube-proxy, cloud-controller-manager and CLOUD PROVIDER API are colored in light blue, while other k8s component and user Pods are colored in dark blue?

/hold

Things to fix:

The spelling is incorrect: kube-api-server should be kube-apiserver;
Use either "Scheduler" or "kube-scheduler", not "scheduler".
If we use binary name for control plane components, then we should say "kube-controller-manager" rather than "Controller Manager".
The color coding is confusing. Why kube-proxy, cloud-controller-manager and CLOUD PROVIDER API are colored in light blue, while other k8s component and user Pods are colored in dark blue?

@utkarsh-singh1 Could you please consider incorporating the review feedback provided above? This would help us advance this pull request.

@tengqm, Cloud-controller-manager, cloud provider API and kube-proxy were considered as useful but optional items in the components architecture that's why they are in light blue coding schemes.

cloud-controller-manager, cloud provider API and kube-proxy were considered as useful but optional items in the components architecture that's why they are in light blue coding schemes.

It's helpful to explain that to a PR reviewer - but it's much more important to make that obvious to readers who are unfamiliar with Kubernetes (for example: add a key to the drawing). Truth be told, website approvers already know these details. We ask this kind of question to highlight that not all readers do.

Some feedbacks to the diagram:
The names of the components in the diagram should be consistent.
You can either call a component by its role, i.e. "API server",
or you can call that component by its binary name, i.e. "kube-apiserver".
We are not supposed to have both "scheduler" and "kube-proxy" in the same diagram.
It is confusing.
Please be precise with the binary name. For example, there is nothing called "kube-api-server".
It is safe to call "kube-proxy" a kubernetes component, although there are solutions
to substitute that with something else in a cluster.
"cloud-controller-manager" is not a must have. It really depends on the infrastructure
that underlies your cluster.
You may want to render "pod"s in a different color.
You may want to have CRI, CSI, CNI side by side on a node, each corresponds to
the computer, storage and network aspect of the runtime environment.
I don't get the reason why "CLOUD PROVIDER API" is capitalized.
How about change that into "Cloud Providers"?

Some feedbacks to the diagram:
The names of the components in the diagram should be consistent.
You can either call a component by its role, i.e. "API server",
or you can call that component by its binary name, i.e. "kube-apiserver".
We are not supposed to have both "scheduler" and "kube-proxy" in the same diagram.
It is confusing.
Please be precise with the binary name. For example, there is nothing called "kube-api-server".
It is safe to call "kube-proxy" a kubernetes component, although there are solutions
to substitute that with something else in a cluster.
"cloud-controller-manager" is not a must have. It really depends on the infrastructure
that underlies your cluster.
You may want to render "pod"s in a different color.
You may want to have CRI, CSI, CNI side by side on a node, each corresponds to
the computer, storage and network aspect of the runtime environment.
I don't get the reason why "CLOUD PROVIDER API" is capitalized.
How about change that into "Cloud Providers"?

@tengqm,

For the components like kube-proxy and cloud-controller-manager , it would be better if they are represented as different colour or marked with some character (*) so that they can be marked as optional components.
As a Kubernetes object , representing Pod in a different color would be a good option.
CRI is already mentioned in the  nodes, CSI and CNI can be mentioned in Nodes specific diagram that can be proposed later.

@utkarsh-singh1  kube-proxy can be replaced, but not removed for most use cases. Without this component, you won't have the Service abstraction. It means you have to manually expose individual Pods to the outside world. This is doable, for example, with Gateway APIs. However, I don't think the community is there already.

kube-proxy can be replaced, but not removed for most use cases. Without this component, you won't have the Service abstraction. It means you have to manually expose individual Pods to the outside world. This is doable, for example, with Gateway APIs. However, I don't think the community is there already.

Even though many clusters still use kube-proxy, you can remove it - and people do. See https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/ for an example.

kube-proxy can be replaced, but not removed for most use cases. Without this component, you won't have the Service abstraction. It means you have to manually expose individual Pods to the outside world. This is doable, for example, with Gateway APIs. However, I don't think the community is there already.

Would it be better if I mark the name like kube-proxy* in the diagram and below mention '* marked components are optional or can be replaced.

kube-proxy can be replaced, but not removed for most use cases. Without this component, you won't have the Service abstraction. It means you have to manually expose individual Pods to the outside world. This is doable, for example, with Gateway APIs. However, I don't think the community is there already.

Even though many clusters still use kube-proxy, you can remove it - and people do. See https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/ for an example.

Yup. This is the only exception I am aware of so far, and I don't think that is a removal. It is a replacement. The Cilium CNI can be configured to manage Service traffics without installing iptables rules.
You will remove kube-proxy only when 1) you have a good understanding about its role in the cluster; 2) you have found an alternative that proxies the service traffic. Instead of putting a component like "kube-proxy or equivalent" into the diagram, I'd suggest we leave it there.

Based on the points.

/label tide/merge-method-squash
/approve

/hold cancel

Hi @tengqm, seems like this PR was closed before I can update the architecture diagram,

should I leave it,
or
Create another PR for the updated diagram, or create another commit on same branch.

Based on the suggestions mention.


clustermesh up/downgrade: test maxConnectedCluster

Configure the max-connected-clusters option for feature coverage in the ClusterMesh upgrade/downgrade tests.
Previously, all clusters in the matrix were using the default value of
255. Now, 2 of the clusters will be configured to support up to 511 clusters. While we are not utilizing the full range of cluster IDs in the test, this feature changes the way identities are allocated and parsed. Enabling this will provide additional test coverage for this aspect.

/test

/test

/ci-gateway-api


quorum-dev-quickstart and azure

Hi there, I am running into problems trying to set up the example on azure. After deploying the package it seems to be stuck. the block explorer isn't registering anything. but the log shows all successes. (see attached) Any idea what's wrong?

I have the same issue.


Disable pip cache

It is useless in docker envs, and just takes space.

FTR: Downstream image may leverage pip cache by:
FROM python:3.10.7-bullseye AS build-pip
ENV PIP_CACHE_DIR=/var/cache/buildkit/pip
RUN mkdir -p $PIP_CACHE_DIR
RUN --mount=type=cache,target=/var/cache/buildkit/pip \
    pip install -U pip wheel

Not exactly. You'll also need:
ENV PIP_NO_CACHE_DIR=

I definitely sympathize with the motivation here (and want this myself for my own FROM python images!), but I do worry about the impact of setting explicit defaults for these, especially when disabling our tweaks to the defaults is (as you've demonstrated) not entirely straightforward/obvious.  Even if this is what most users actually want, it is a surprising moment when they discover that the defaults they thought applied (and that the pip documentation describes applying!) actually do not seem to apply in practice. :sweat_smile:

I understand your concern. I'll try to check what other popular image do wrt caching.

Ok, you're probably right. I also see that it is a common practice to use a mount for the cache, which will be broken if this change is accepted.
So maybe we should just recommend setting this variable in the README instead?

Actually I see that the documentation of the image already uses --no-cache-dir in the example, so I guess no further action is needed:
RUN pip install --no-cache-dir -r requirements.txt

(notice that this affects only the packages that are installed during the build, and has no effect when running pip in the container, but this is less important).
Also this doesn't cover packages that require build. See https://github.com/pypa/pip/issues/12031


on http://tour.dlang.io/tour/en/basics/associative-arrays

also here maybe similar stuff as in ruby hash and even more details from std library
(reported by @claudiug)

@claudiug what exactly do you miss?

more example for std.library
more example and how to use them properly. aka, add, remove, search, find.
example: what is the return value of byValues or byKeys
get(key, defaultValue is this simillar with ruby: .fetch('foo', []). where [] is what I will return is null/nill is there?

thanks for clarifying - I tried to enhance the explanation in #374.

more example for std.library

Afaik only std.array provides specific methods for associative arrays, so not much so say there.
However std.array is mentioned now.

more example and how to use them properly. aka, add, remove, search, find.

search and find can both be done with in (does this get clear now?)
add = assignment
remove was already mentioned at the bottom of the page (I tried to highlight it a bit)

example: what is the return value of byValues or byKeys get(key, defaultValue is this simillar with ruby: .fetch('foo', []). where [] is what I will return is null/nill is there?

tried to rephrase get explanation
added a small example for byValue at the bottom ;-)


Build srcs that are other build targets

puku seems to remove srcs that are not existing files. Does this PR solve this problem too?

I'm not sure if it does actually. I'll add a test for that next week.

Okay, can confirm this fixes that issue :)

@Tatskaari
I've tried this PR in my environment and found that:

using variables in srcs seems to not work, like using codegen = text_file()
output files that are in a provides: {"go": ":label"} of a label in srcs seems not used

using variables in srcs seems to not work, like using codegen = text_file()

Yeah this is best effort. I want to write a proper evaluator but I think i'm going to need to become persuasive to get that into the roadmap haha

output files that are in a provides: {"go": ":label"} of a label in srcs seems not used

Ooff yeah that's a good point. I will have to think about this. Thanks for the feedback though.

Yes, in my use case, I use a proto_library() to only generate go files then use them and other non generated go files in a go_library().

Ah, fair enough. I've raised an issue here:
#62
however I think this might be better off living in https://github.com/please-build/buildtools


Bitmaps using Array

Store values as bitmaps of row indexes.  Treat a generic JavaScript array as a packed array of 32-bit integers, and see what can be done to hint at the JavaScript runtime that it should be stored as such.
This is a stepping stone to using Uint32Array.  It's also a fallback for older browsers, and may be faster for filtering.

Jordan finished this a year ago;  currently in active use.


Excluding a specific RuboCop cop on a specific haml-lint file

How do I exclude a specific RuboCop cop on a specific haml-lint file?
I tried adding the haml file to a specific Cop's Exclude in .rubocop.yml, but that didn't help.
I could disable the cop for haml-lint (using RuboCop ignored_cops in .haml-lint.yml), or disable RuboCop for a specific file (using RuboCop exclude in .haml-lint.yml), but I'd prefer to do something more specific. If need be, I could use inline exclusion of RuboCop, but that's less than ideal.

The reason excluding the HAML file itself in your RuboCop config doesn't work is because HAML-Lint actually converts the .haml template into a temporary .rb file under the hood, so the names won't match. I don't think there's any clever way around this since the generated name is intentionally random to avoid conflicts.
It's not great, but you should be able to add a comment in Ruby code in your HAML file:
- # rubocop:disable Lint/UselessAssignment
- unused_variable = 32

Notice that the comment is actually a script with just a Ruby comment. It is not a HAML comment (i.e. -#). The two are treated differently. HAML comments never get compiled into Ruby (which is why the  RubyComments encourages you to use them).
However, if you find yourself needing this often, disabling RubyComments may make sense so that you can disable RuboCop cops for a specific file within the file itself.
Another approach would be to declare all the cops you want to disable in this specific HAML template at the top of the template within a :ruby filter:
:ruby
  # rubocop:disable Lint/UselessAssignment, Metrics/PerceivedComplexity, etc.
%p
  My template

This stands out a bit better, making it clear which cops you've disabled for this specific file. It also saves you from having to disable the RubyComments linter.
Hope that helps!


DiagOFTModule requires einops package

Installing lycoris-lora module and a file with from lycoris import DiagOFTModule errors about missing einops package. Additionally requires toml module (but maybe more generally than just DiagOFTModule).
I was doing some independent testing and wanted to mention this. Thank you.

Thx

Added in dev


Private Link Swagger Change

MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Changelog
Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?

[ ] new service onboarding
[ ] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[x] Other, please clarify
Deprecate PrivateLinkConnectionApprovalRequest and use RemotePrivateEndpointConnection as the request type of Approval API to meet [ARM requirements](OAPI021: PUT/PATCH shouldn't fail for readonly properties).

When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.  API changes are already deployed in public regions.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month. With ANT97.
If updating an existing version, please select the specific language SDKs and CLIs that must be refreshed after the swagger is published.

[ ] SDK of .NET (need service team to ensure code readiness)
[ ] SDK of Python
[ ] SDK of Java
[ ] SDK of Js
[ ] SDK of Go
[ ] PowerShell
[ ] CLI
[ ] Terraform
[x] No refresh required for updates in this PR

Contribution checklist:

[x] I commit to follow the Breaking Change Policy of "no breaking changes"
[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

Applicability: :warning:
If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs
Adding new properties
All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

[x] Check this box if any of the following apply to the PR so that label "WaitForARMFeedback" will be added automatically to begin ARM API Review. Failure to comply may result in delays to the manifest.

Adding a new service
Adding new API(s)
Adding a new API version
-[ ] To review changes efficiently, ensure you are using OpenAPIHub to initialize the PR for adding a new version. More details, refer to the wiki.

[x] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[x] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If any of the following scenarios apply to the PR, request approval from the Breaking Change Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in a stable version
[x] Removing properties in a stable version
[ ] Removing API version(s) in a stable version
[ ] Updating API in a stable or public preview version with Breaking Change Validation errors
[ ] Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.
Please follow the link to find more details on PR review process.

Add SDK owner on schema name change, which likely causes SDK breaking
@msyyc @qiliao123 @ArcturusZhang @tadelesh


Add possibility for e-current

Pull Request Test Coverage Report for Build<PHONE_NUMBER>1
Details

2 of 3 (66.67%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 99.584%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

empymod/model.py
2
3
66.67%

Totals

Change from base Build<PHONE_NUMBER>:
-0.03%

Covered Lines:
3109

Relevant Lines:
3122

💛  - Coveralls


export web dashboard

please, change html name to dashboard.html

please, change html name to dashboard.html

done, can you review the changes?


Update boost to 1.69. Fixes #5023

Remove boost-signal (rationale: deprecated and removed in 1.69)
Remove boost-hana workaround by xiangfan-ms #4186 #3579 (rationale: 1.69 incorporates required fixes in hana master branch; builds fine in VS2017 15.9)

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: McDamon sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Renaming integrations for consistency

Pre-requisites
Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[x] If my change requires a change to the documentation, I have updated the documentation accordingly.
[x] I have added tests to cover my changes.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Describe changes
Changing the name of the sklearn integration into scikit-learn integration
Please let me know whether this change makes sense to you and whether the pytorch integration must be renamed to the torch integration in your opinion.

Initial feedback from me on the scikit-learn stuff that's in there in this draft. I find it quite confusing to know when a hyphen is used in between the scikit and the learn vs when an underscore is used. I think if anything it should be standardised to a hyphen, since you never see scikit_learn out in the world (though I could well be wrong on that). I don't have a strong opinion on torch except that whatever we do it should be clear / consistent in our docs.

Should we close this PR @bcdurak ?


Node exporter instance by hostid

Adding a dashboard with the node_exporter metrics.
This patch only adds the metrics. After we'll be happy with the results a followup patch will remove those metrics from other dashboards.

In the templating parameters, I suggest moving the node selection to the left.
Does not make sense between disk and metwork

I couldn't find a way to limit the panel width, it gets smaller as you add more panels, but I agree it does look strange.
I'll change the template parameter order

Maybe you can limit the row width?
See  https://github.com/grafana/grafana/pull/8199 (not solved)

@amnonh can you please merge with #320 so this can be tested?
(or suggest another way to test the master dashboard)

@amnonh ping

Its better to place the node selection to the most left or the most right (I think left is better)

@amnonh the per machine dashboard looks good

With it, we can remove the metrics from the per server dashboars (disk, network etc)

@glommer can you take a look before we merge?

Looks good to me.

Amnon can you please address the name comment above
https://github.com/scylladb/scylla-grafana-monitoring/pull/318#issuecomment-387830750

I already replaced the name, do you see something else?

I already replaced the name, do you see something else?


Crashes on setup, Android 5.0

I just tap on set up account and it crashes.
Samsung Galaxy S5

Dup of https://github.com/google/google-authenticator-android/issues/30


Buggy example - yet: this is amazing!

Hi, your example doesn't work when using tuples:
gps_ifd = {
pyxif.GPSIFD.GPSVersionID: 1,
pyxif.GPSIFD.GPSDateStamp: "1999:99:99 99:99:99",
}
Traceback:
File "pyxif_load_and_dump.py", line 373, in dict_to_bytes
value_str = raw_value + b"\x00" * (4 - length)
TypeError: can only concatenate tuple (not "str") to tuple
It works when using pyxif.GPSIFD.GPSVersionID: "1" -> string instead of int as value.
Apart from that, your lib looks very promising! If this works, we'll make use of it on Pixabay.com and we'll certainly write about it on our blog :-) Keep up the great work!
It clean and simple and doesn't require unreliable third party libs, such as pyexiv2 ...

I've narrowed the issue down a bit. I run into trouble when using Short and Byte values.
E.g. assigning a Flash value works when using a tuple instead of an integer:
pyxif.ExifIFD.Flash: (1,), # Short
I couldn't get any Byte value to work, such as:
pyxif.GPSIFD.GPSAltitudeRef: 1
Both works well with EXIF, your pure PIL helper.

E.g. assigning a Flash value works when using a tuple instead of an integer:

Tuple was implemented for any length value. Sometimes BYTE, SHORT, and LONG take two or more length value.
I'm thinking that the value should be checked length or not.

couldn't get any Byte value to work

Fixed now,
Thank you.

Comments and suggestions are welcome : )

I think tuples are fine, but it would be helpful for others if it were mentioned in the example code.
Anyways, as of this morning, we already are using your other app, "EXIF", in production on Pixabay. And it works really well so far! We've replaced the outdated app "pyexiv2" with EXIF in combination with Pillow.
If you can, write some more examples and/or a short documentation. Post it on Stackoverflow as an answer to relevant questions. There are people out there looking exactly for what you've created here. And it was really hard to find :-P
We'll keep on testing it on Pixabay ...

It's my pleasure that my code is used by Pixabay useful, cool and famous global web service.
I wrote "EXIF" to resolve an issue in Pillow. I don't know that I will keep developing "EXIF".
At least, I think I'll keep developing "Pyxif", so I recommend to use "Pyxif".

I made a documentation and wrote tuple matter there. I'll close this issue.
Ver. 1.0.0 document: http://piexif.readthedocs.org/en/1.0.x/appendices.html#exif-data-in-piexif

It's a great start! If I find the time some day in the future, I'd definitely like to help you with this project.


Report: The top X populated cities in a country where X is provided

A report which need to contain the following columns

Name.
Country.
District.
Population.

provide a Report on The top number of  populated cities in a country where the number is provided.
e.g. provide a report on the top 5 populated cities within a country.


2019-11-29-0010-I need to update my spark repository.

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.

Just do it .


Expose Wallet ID through refactor

The core vegawallet has a concept of wallet ID which is  the public key for the internal root node at the 'VEGA point. This however requires a bit of deviation from the current implementation  which hides the internal derivation nodes

Fixed in https://github.com/vegaprotocol/js-crypto/commit/89d0c2e585b0494999a37ee423945726c63b62aa


tests: refactoring the custom PCI addressing test suite

Custom PCI configuration tests were improved in the following manner:

PCI addresses were assigned to all the disks that are atthached to the VM.
Previously, only additional disks were given a manual PCI address while the pre-defined ones left
w/o assignment, leaving it to libvirt for automatic assignment
Explicit boot order was defined on the bootable VM disk.
Cosmetic changes.

Signed-off-by: Igor Bezukh<EMAIL_ADDRESS>What this PR does / why we need it:
When configuring manual PCI address for a VM devices
it must be done for all the devices. Otherwise an inconsistent
behavior may occur.
One of the examples is when not all the block devices are
given a manual PCI address. In such a case the BIOS may choose
to boot the disks according to the PCI mapping and the bootable disk
may not be the first, thus causing the VM to hang on boot (even if
specify the boot order explicitly in the VM spec).
Which issue(s) this PR fixes :
https://bugzilla.redhat.com/show_bug.cgi?id=1888720

/cc @vladikr

/cc @vatsalparekh

/retest

/retest

/lgtm

/cherry-pick release-0.34


is there any way to apply border colour to chat bubble and set nip to centerleft or centerright?

to set nip to centerleft or centerright i use "nipOffset". Don't know how to apply border colour?

Sorry. I'm making a new version now. Unfortunately, there is not enough time. I will do it.


Twitter unsupported URL error

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2020.03.24
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

youtube-dl.exe -v "https://twitter.com/VictorKvert2008/statuses/1251621111357456384"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://twitter.com/VictorKvert2008/statuses/1251621111357456384']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2020.03.08
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: none
[debug] Proxy map: {}
[generic]<PHONE_NUMBER>357456384: Requesting header
WARNING: Falling back on generic information extractor.
[generic]<PHONE_NUMBER>357456384: Downloading webpage
[generic]<PHONE_NUMBER>357456384: Extracting information
ERROR: Unsupported URL: https://twitter.com/VictorKvert2008/statuses/1251621111357456384
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp378onz6j\build\youtube_dl\YoutubeDL.py", line 797, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp378onz6j\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp378onz6j\build\youtube_dl\extractor\generic.py", line 3351, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://twitter.com/VictorKvert2008/statuses/1251621111357456384

Description
Can't download some videos from Twitter (unsupported URL error).

Could you try with this url? https://twitter.com/i/status/1251621111357456384

you need only the id so this regex is better:
https?://(?:(?:www|m(?:obile)?)\.)?twitter\.com/(?:(?:i/web|[^/]+)/(?:status|statuses))/(?P<id>\d+)
https://regex101.com/r/Fv38pg/6
regards Oneboy1979

Could you try with this url? https://twitter.com/i/status/1251621111357456384
Yes, this URL works. Thanks!


auto Blue Field Front

This issue will include the tuning to get the robot to each of the spike lines.


[CI] Refactor Jenkins CI pipeline + migrate tests to Jenkins

See #4234 for details.
Overview of Jenkins CI pipeline:

Will re-open after filling in all blanks

@CodingCat I'm seeing this error with the latest master:
19/04/17 12:19:38 ERROR XGBoostSpark: when specifying eval sets as dataframes, you have to ensure that the number of elements in each dataframe is larger than the number of workers

https://xgboost-ci.net/blue/organizations/jenkins/hcho3-xgboost-ci-test/detail/refactor_jenkins/71/pipeline/33#step-68-log-2079
How serious is this error? Also, do we want to build a XGBoost JAR with one version of Spark and test it with another version of Spark?

For the time being, I took out all changes related to pylint.

Oh, I didn’t notice the question here, I think it’s because we have too many cores in jenkins?
How many cores we have there?

@CodingCat 8 cores

it’s strange...is the issue still there?

@CodingCat For now, I am skipping JVM tests from the Build stage. Trying to export a test JAR so that I can run JVM tests in the Test stage. See https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-4380/18/pipeline/173

@CodingCat What's the meaning of this error?
- repartitionForTrainingGroup with group data which has empty partition
Tracker started, with env={DMLC_NUM_SERVER=0, DMLC_TRACKER_URI=<IP_ADDRESS>, DMLC_TRACKER_PORT=9091, DMLC_NUM_WORKER=16}
19/04/24 02:52:58 ERROR Executor: Exception in task 10.0 in stage 2.0 (TID 17)
java.util.NoSuchElementException: key not found: train
        at scala.collection.MapLike$class.default(MapLike.scala:228)
        at scala.collection.AbstractMap.default(Map.scala:59)
        at scala.collection.MapLike$class.apply(MapLike.scala:141)
        at scala.collection.AbstractMap.apply(Map.scala:59)
        at ml.dmlc.xgboost4j.scala.spark.XGBoost$.ml$dmlc$xgboost4j$scala$spark$XGBoost$$buildDistributedBooster(XGBoost.scala:148)
        ...
9/04/24 02:52:58 ERROR Executor: Exception in task 9.0 in stage 2.0 (TID 16)
java.util.NoSuchElementException: key not found: train

Will re-open when things are ready. There's a long way to go

@CodingCat FYI, reducing the number of cores did the trick. Thanks for the suggestion

@hcho3, yeah, it should work, but 8 cores should not make the failure happen, I met it in a 40-cores machine...
Partitioner of spark may have some secretes that I don’t know...I will check


tools: make upgrade tool copy docs from go-mysql-server

Fixes #793
There are some docs we want to have in gitbase as well as
go-mysql-server. To avoid possibly having different versions in
each project, this will automatically copy certain parts and files
from one to the other when there is an update.

Should be fixed now

Truly fixed now

Upgraded to latest go-mysql-server


Example not running

When I run the code under Example I am getting this error:
UndefVarError: FixedWidth not defined

Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base.jl:26
 [2] top-level scope
   @ In[1]:4
 [3] eval
   @ ./boot.jl:360 [inlined]
 [4] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
   @ Base ./loading.jl:1094

Yes, that made it work. I am closing the issue, thanks again for your help!


Add pickling methods to support TorchScript.

Sphericart already uses TORCH_LIBRARY but when serializing the library reference into a TorchScript JIT module, the class needs a pickle/unpickle method as well.

Thank you for doing this! The code looks good but fails the formatting test. Could you run clang-format on the files you modified? Then everything should be good to go!

Could you run clang-format on the files you modified?

Sure! I tried this quickly and it reformatted the entire file, which probably means I did something wrong. I'll look into it shortly.

Thanks for the PR, @HapeMask! Could you share with us the exact clang-format command you ran (and also the directory you ran it from)? We're trying to improve our C++ formatting, perhaps I can open a PR for that

Yup, here's some info on what I ran + version:
]$ clang-format --version
clang-format version 16.0.1 (https://github.com/conda-forge/clangdev-feedstock 1dd6d282bc8be5828cf8c163d2c68cc525f527c5)

]$ clang-format -i sphericart-torch/src/torch.cpp

]$ git log
commit 2f07d5f (HEAD -> pickle, mine/pickle)
Author: Gabe Schwartz<EMAIL_ADDRESS>Date:   Wed Oct 25 15:52:31 2023 -0700

    Add pickling methods to support TorchScript.

commit 8ffaf38 (origin/main, origin/HEAD, main)
Author: Filippo Bigi<EMAIL_ADDRESS>Date:   Fri Aug 18 14:20:46 2023 +0200

    Fix computations for large output sizes (#67)

    * Fix large-size integer overflows on CPU
    * Allow computations on large tensors on GPU

    ---------

    Co-authored-by: Nick J. Browning<EMAIL_ADDRESS>    Co-authored-by: Guillaume Fraux<EMAIL_ADDRESS>

and this is the changeset it produced for that file: https://gist.github.com/HapeMask/31d530a691b0937344d7392eee8cc0e0
I'm happy to just copy/paste the formatting changes to only the code I added, just let me know.

I tested clang-format-10 and clang-format-17 on main and they give the exact same results, so I would guess that version is not the issue. From the changelog that @HapeMask sent, it seems that the first indent is 2 spaces, while all others are 4 spaces. It's also reordering the includes (alphabetically) and it seems to dislike long lines. Almost as if it was operating according to a different set of formatting instructions. @Luthaf any ideas? What should we do?

What happens if you run clang-format --style=file:$(pwd)/.clang-format -i sphericart-torch/src/torch.cpp? I though this was the default behavior, but let's check!

Oh! I don't know how this happened but my clone of your main branch was from Aug. 18th (before the .clang-format file was added to the repo, so I didn't have that file in my branch. Re-running after merging the latest main branch gives better results, I'll push the changes today.


[Android] Hand Control Scrolling not working on Flutter 2.8.1

This is not happening until I upgraded Flutter framework version of my project from 2.2.1 to 2.8.1.
FYI, it happened on android devices, with Cobrowse SDK version 2.19.0 and I have tried updating the Cobrowse SDK version to the latest which is currently 2.23.0 but no luck.
I have tried to reach out to the Cobrowse Support team and they think the issue could be the flutter framework upgrade. Since they are not officially support for flutter app, so they actually mentioned that I can report the issue here.
Please advise. Thanks!
You can refer to the demo video here.

FYI, this issue only happened in Flutter UI, meaning that native UI from Java activity was totally working fine.

@robinbonnes

Hello @chengboonrong,
I'm on the latest Flutter stable version (3.3.4) and here it is working for a normal listview menu:
 @override
  Widget build(BuildContext context) {
    return Drawer(
      child: ListView(
        children: <Widget>[]
      ),
    );
  }

Maybe its due to any custom scroll gesture?

Hi @robinbonnes , it seems to be working on Flutter version 3.0.0 onwards again. Thanks for the update here!


update build_backend to be compliant with PEP517

Allow to use only pip install -e . --no-deps in the docker image without changing the poetry build system.
see documentation : poetry and pep 517
(Hello!, I am the freelance software engineer hired by Hector to build you and your team a jupyterlab docker image with all the tools you need for development.)

Hi Marine,
do you want me to add you to the project so you can have full access?

Hi Katharina, yes, that's could be practical.
I will add some parallelization feature for the decoding lib. I can modified your decoding notebook to show you how to speed up your code.
But the most urgent right now is to merge this PR.
Thank you
Have a good day

Can you merge the pull request yourself with the access to the repository?

Yes I probably can but it is an unusual request. You don't want to verify what i merge on your project ?
I would advise for my next PR you check what I change before it being merged.
I am merging this one as you gave implicit approval in your previous comment.
Thank you have a good day

I don't work on this project anymore (and haven't done so for over a year). Hector intents to use this work for other data and future students. I am happy to check whenever I can, but at the moment I am in the middle of a move and cannot check on a very regular basis. So if this is delaying your work, I am happy for you to do the merges.


Clarification about some options

I can't find any details about what are the specific operations done in "advanced" mode.
Can this be explained in details? I want to know if it's safe to do this with my CSS.
Also, what is "restructuring" and "aggressiveMerging"?
Can I see examples about what these operations actually do?

There's no manual for it but as they say, "use the force, read the source" :-) So here we go:
All advanced operations are dispatched here: https://github.com/jakubpawlowicz/clean-css/blob/master/lib/selectors/advanced.js#L59

recursivelyOptimizeBlocks calls all following operations on a block (think @media or @keyframe rules)
recursivelyOptimizeProperties optimizes properties in selector rules and "flat blocks" (like @font-face) by splitting them into components (e.g. margin into margin-(*)), optimizing, and rebuilding them back. You may want to use shorthandCompacting option to control whether you want to turn multiple (long-hand) properties into a shorthand property.
removeDuplicates gets rid of duplicate selector rules with exactly the same properties (think of including the same Sass / Less partial twice for no good reason)
mergeAdjacent does what the name implies for same adjacent selectors
reduceNonAdjacent figures out which properties are overridden for same non-adjacent selector rules
mergeNonAdjacentBySelector figures out if non-adjacent selector rules can be moved (!) to be merged, requires all intermediate rules to not redefine the moved properties, or if redefined to be either more coarse grained (e.g. margin vs margin-top) or have the same value
mergeNonAdjacentByBody does what the one above but by searching for two rules with same properties rather than with same name
restructure tries to reorganize rules with different names and properties so they take less space, e.g. .one{padding:0}.two{margin:0}.one{margin-bottom:3px} into .two{margin:0}.one{padding:0;margin-bottom:3px}
removeDuplicateMediaQueries this one is easy :-)

aggressiveMerging cares about property order when merging, e.g. if it's turned on then a{display:block;color:red;display:inline-block} will be turned into a{color:red;display:inline-block} - if two display properties were next to each other it would keep them as is.
If your content is not-so-well structured and / or uses some edge CSS stuff I'd turn aggressiveMerging, restructuring, and shorthandCompacting off as these may break things.
You can find more tests here: https://github.com/jakubpawlowicz/clean-css/tree/master/test/selectors
Hope this helps.

I reworded my post and added it to readme: https://github.com/jakubpawlowicz/clean-css#what-advanced-optimizations-are-applied

@jakubpawlowicz This is great, thanks!

Are all of these advanced optimizations 100% safe?
Are they safe if you have external CSS that's not compiled by CleanCSS?
If not, are any of them specifically more or less safe than the others? I'd be interested in learning about any nuances here.

Sorry for a bit of a lag with the reply.
I would call all but restructuring and shorthand compacting safe. Those two should be as well but there could be some edge cases, and question of safety also depends on the context, as if you target some old browsers or email clients you can expect those to break compatibility.
I am currently thinking about standardizing those "safety levels" in the next version of clean-css.


feat: hover to activate a tab

In case we use the tab inside a mega menu, we might need the ability to activate a tab via mouse hovering instead of just clicking.

Close as not really in need at the moment.


Does this issue occur when all extensions are disabled?: Yes

[ ] @[

[](url)

](url)

I can confirm this, it is quite annoying. I believe the root cause is that the git pane refreshes when a commit is staged, but if the cursor is active in the commit message box it ceases to be active there when the panel is refreshed.

Duplicate of https://github.com/microsoft/vscode/issues/92401


Wrong state for popup menus?

When you hover over a popup menu item, the state is set to "focused", not "hovered". I'm unsure if this is a bug or intended; probably need some more testing to make sure,

It looks like a real bug. As a result, each menu item forever stuch it a hovered state after the first hover


Provide versions of Twirp and GrpcWeb transport that don't require fetch API polyfills

The Twirp and GrpcWeb transports require polyfills for the fetch API on node.
React Native includes a polyfill for the fetch API , but it is incomplete, see #67.
Instead of relying on fetch API polyfills, it might be a better alternative to provide two new packages grpcweb-node-transport and tiwrp-node-transport which use only native node APIs for HTTP.
The downsides are added maintenance and more choices for packages to understand.

This issue has been alleviated by #81.
grpcweb-transport is now compatible with node-fetch.

This issue has been alleviated by #81.
grpcweb-transport is now compatible with node-fetch.

Closing this for now because of alleviation by #81.


adaptive concurrency: Trigger minRTT calculation outside interval

This patch changes the adaptive concurrency filter's minRTT measurement behavior to additionally trigger a minRTT calculation if there have been 5 consecutive concurrency limit updates at the minimum concurrency. Previous minRTT behavior is still maintained, but the new condition resets the timer on the minRTT calculation.
The minRTT convergence times should be improved for little to no observable differences in filter behavior.
Risk Level: Low
Testing: Unit tests
Docs Changes: yes
Release Notes: yes

Can you take a look at clang-tidy errors?
/wait


Imagery

From @joebayles on September 28, 2015 18:37
https://nsec.esri.com/arcgis/home/item.html?id=55a20dc19f2e47c8ad3b0a38eca94671

[ ] Title?
[ ] No Access and Use Constraints
[ ] Delete Protection Disabled

Copied from original issue: ArcGIS/mil-intel-desktop-solutions-templates#133

Fixed based on AGOL. Back at you @joebayles

Released to ReleasedOpsServer


Peer dependency version

Hello there, would you mind improving the peer dependency specification of the esbuild-node-externals package? It currently specifies
esbuild: ^0.13.0 which leads to yarn complaining when upgrading esbuild to v0.14. The reason is because the ^ only allows newer versions that do not increment the first non-zero portion of the version. In your readme you write that esbuild from version 0.8 is supported, so I suggest something like esbuild: >0.8.
Or probably better if you want to check for breaking changes with each release that bumps the minor version counter you could specify a range: esbuild: 0.8 - 0.13.
If you want to play around with the version specifier, there's this site: https://semver.npmjs.com/

Good point, actually with the latest feature it should probably be 12+, happy to accept a pr to fix it :)

Cool, coming right up

Amazing, thanks!


feat: further theory about the Killing form of Lie algebras

[x] depends on: #6588

bors merge


[models] Email: allow NULL but set UNIQUE constraint #28

NULL does not trigger the UNIQUE constraint so we
can allow the system to have users without email
(for users automatically generated by the radius module)
Related to #28.

Coverage decreased (-0.3%) to 96.061% when pulling b28ab212be9199ac37ff8bdb94a6fc5e43a109d8 on unique-email into fb7b96f4ff162a75e5d6818b126e3887abde9f00 on master.


flutter create shows a long list of added pub dependencies, even though the user hasn't added anything yet

Most of the output from flutter create is showing the pub dependencies that are added from every blank template.  Suppress this on an initial create, it adds a lot of noise, and the user hasn't explicitly added these dependencies so it's confusing and hiding the actually interesting "how to run" instructions.
Similar request to https://github.com/flutter/flutter/issues/93036.
Probably caused by https://github.com/flutter/flutter/pull/110851?
$ flutter create test_create
Creating project test_create...
Running "flutter pub get" in test_create...                        997ms
Resolving dependencies in test_create...
+ async 2.10.0
+ boolean_selector 2.1.1
+ characters 1.2.1
+ clock 1.1.1
+ collection 1.17.0
+ cupertino_icons 1.0.5
+ fake_async 1.3.1
+ flutter 0.0.0 from sdk flutter
+ flutter_lints 2.0.1
+ flutter_test 0.0.0 from sdk flutter
+ js 0.6.5
+ lints 2.0.1
+ matcher 0.12.13
+ material_color_utilities 0.2.0
+ meta 1.8.0
+ path 1.8.2
+ sky_engine 0.0.99 from sdk flutter
+ source_span 1.9.1
+ stack_trace 1.11.0
+ stream_channel 2.1.1
+ string_scanner 1.2.0
+ term_glyph 1.2.1
+ test_api 0.4.16
+ vector_math 2.1.4
Changed 24 dependencies in test_create!
Wrote 127 files.

All done!
You can find general documentation for Flutter at: https://docs.flutter.dev/
Detailed API documentation is available at: https://api.flutter.dev/
If you prefer video documentation, consider:
https://www.youtube.com/c/flutterdev

In order to run your application, type:

  $ cd test_create
  $ flutter run

Your application code is in test_create/lib/main.dart.

Reproducible only on master.

logs
nexus@Daniels-MBP nevercode % flutterm create n
Creating project n...
Running "flutter pub get" in n...                                2,065ms
Resolving dependencies in n...
+ async 2.10.0
+ boolean_selector 2.1.1
+ characters 1.2.1
+ clock 1.1.1
+ collection 1.17.0
+ cupertino_icons 1.0.5
+ fake_async 1.3.1
+ flutter 0.0.0 from sdk flutter
+ flutter_lints 2.0.1
+ flutter_test 0.0.0 from sdk flutter
+ js 0.6.5
+ lints 2.0.1
+ matcher 0.12.13
+ material_color_utilities 0.2.0
+ meta 1.8.0
+ path 1.8.2
+ sky_engine 0.0.99 from sdk flutter
+ source_span 1.9.1
+ stack_trace 1.11.0
+ stream_channel 2.1.1
+ string_scanner 1.2.0
+ term_glyph 1.2.1
+ test_api 0.4.16
+ vector_math 2.1.4
Changed 24 dependencies in n!
Wrote 127 files.

All done!
You can find general documentation for Flutter at: https://docs.flutter.dev/
Detailed API documentation is available at: https://api.flutter.dev/
If you prefer video documentation, consider: https://www.youtube.com/c/flutterdev

In order to run your application, type:

  $ cd n
  $ flutter run

Your application code is in n/lib/main.dart.

flutter doctor -v
[!] Flutter (Channel master, 3.6.0-12.0.pre.20, on macOS 13.0.1 22A400 darwin-arm64, locale en-GB)
    • Flutter version 3.6.0-12.0.pre.20 on channel master at /Users/nexus/dev/sdks/flutters
    ! Warning: `flutter` on your path resolves to /Users/nexus/dev/sdks/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/nexus/dev/sdks/flutters. Consider adding /Users/nexus/dev/sdks/flutters/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/nexus/dev/sdks/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/nexus/dev/sdks/flutters. Consider adding /Users/nexus/dev/sdks/flutters/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 02de12947a (9 hours ago), 2022-11-29 22:57:31 -0500
    • Engine revision d5690468da
    • Dart version 2.19.0 (build 2.19.0-443.0.dev)
    • DevTools version 2.20.0
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at /Users/nexus/Library/Android/sdk
    • Platform android-33, build-tools 33.0.0
    • Java binary at: /Users/nexus/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/213.7172.25.2113.9123335/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14B47b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[!] Android Studio
    • Android Studio at /Applications/Android Studio Preview.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    ✗ Unable to find bundled Java version.
    • Try updating or re-installing Android Studio.

[✓] Android Studio (version 2021.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[!] Android Studio
    • Android Studio at /Applications/Android Studio Preview 2.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    ✗ Unable to find bundled Java version.
    • Try updating or re-installing Android Studio.

[✓] Android Studio (version 2021.3)
    • Android Studio at /Users/nexus/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/213.7172.25.2113.9123335/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] Android Studio (version 2021.3)
    • Android Studio at /Users/nexus/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/213.7172.25.2113.9014738/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] VS Code (version 1.73.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.52.0

[✓] Connected device (4 available)
    • M2007J20CG (mobile)    • 5dd3be00                             • android-arm64  • Android 12 (API 31)
    • iPhone 14 Pro (mobile) • 4F72110C-F38B-4CF9-93C4-4D6042148D28 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-16-1 (simulator)
    • macOS (desktop)        • macos                                • darwin-arm64   • macOS 13.0.1 22A400 darwin-arm64
    • Chrome (web)           • chrome                               • web-javascript • Google Chrome 108.0.5359.71

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 3 categories.

@sigurdm

I guess we can suppress the output of flutter pub get run from flutter create if it is too much.
We internally in pub get have a "quiet" mode (only reports success or failure), and could perhaps expose and use that.
@jonasfj WDYT?

@jonasfj suggests that we activate the quiet (one line summary) behaviour in pub depending on some environment variable.
Perhaps we can even look at the PUB_ENVIRONMENT that is set by flutter_tools.

I think it's okay to have on flutter pub get but not flutter create

This code already supports not inheriting stdio for tests, we should also use that for the flutter create workflow. cc @andrewkolos

Is there any way we can suppress Resolving/Got lines as well?
Resolving dependencies in test_thing...
Got dependencies in test_thing.

Is there any way we can suppress Resolving/Got lines as well?
Resolving dependencies in test_thing...
Got dependencies in test_thing.

#118285, which closed this issue, uses the recently-added PUB_SUMMARY_ONLY env var, which still prints the two lines you mentioned. This was used for its simplicity and it avoids the vast majority of the noise generated by pub get during flutter create.
Regarding the two lines you mentioned, I imagine it's possible to avoid them by instead having pub get not inherit stdio from flutter create (it would still listen on stderr for the sake error reporting). However, I am personally unsure that it's worth the development effort or code complexity to do so, only because I don't find these two lines are very noisy. I'm interested in any further thoughts


Add space option to apps:create

This adds a hidden --space option to the apps:create command as a first step for moving this off the dapps namespace. This still uses a variant until this endpoint is in the mainline API, but we have gotten enough negative feedback about having to use dapps:create to create apps in spaces that it is important to at least have this command in the mainline CLI.
The only other command under dapps is dapps:info, which will be done separately because 1) it will require API changes and 2) it is less important.
/cc @tim-lang

Similar to listing, should there be a short-arg for space here? (this one seems fine, might be good to add that though).

should there be a short-arg for space here

The option short-arg (-s) is already used by --stack and didn't really want to add a short args for a hidden option. Perhaps we could use capital -S (thoughts?), but again, that's something I'd probably not want to add until this is a visible arg.

Not urgent, just seemed inconsistent compared to the other args. Certainly can wait.


support save dir argument

I think it will be better if labelImg can provide an extra saveDir argument in command line, for convenience to script everything, like this:
python labelImg.py [IMAGE_PATH] [PRE-DEFINED CLASS FILE] [SAVE DIR]

I'm working on it.

thanks


Illidan didn't play his dialogue when completing the Seat of Power quest

Closed by https://github.com/AzerothWarsLR/WarcraftLegacies/commit/8277b6910c9fdfc37635cd543eb4c4c42147ba58


Serialisation options

Hey, just wondering how strongly you're tied to bincode?
I've been using bitcode in my project and it has nice size benefits over bincode but it does use it's own serialisation traits, not the ones from serde. Theoretically.. would you be open to a PR for an optional feature that enabled the use of bitcode?

I am not strongly tied to bincode. It was just a good "by default" option.
In fact I still had plans to investigate a switch to postcard as a default serializer/deserializer and, in the process, see how to allow for more custom defined ones. As such, I am not against a PR that would add bitcode support, it seems like a great alternative.


Small typo fix

Whoops, didn't see this PR until after I had already just submitted a fix directly to develop. Thanks!


Card Capture & Animation

Flipping animation to be handled in subsequent version of this app.
Test backgroundColor change further increase efficiency and allow for future customization.

Implement as design variable in subsequent version of this project.


WIP: Cache integration tests data in datagen level

A faster solution for #6201 , still WIP.
PR #8441 cached dataframes when generating integration tests, and after commits the performance bottleneck for IT seems to be sre_yield.
In this PR, I'm trying to cache DataGen.gen to reuse more data, so that many costly operations like string generation can  be saved.
I thought it was a simple change, but my code always breaks some test cases, and as soon as I fix one, some others break. I started to doubt if this solution was right, so I draft this PR for some early review.
This PR:

Cached DataGen.gen, added a step parameter for it, and updated all related IT cases.
Added _cache_key in DataGen to avoid multiple calls of _cache_repr.
Added a rand object and seed inside Datagens.

Note:

This results may not be the same between cached / not cached even if the seed and other parameters are the same, it is because the gen have side effects on the rand. But I think it will be consistent in dataframes level.
DataGen with same parameter and same seed will generate same output now, no matter it is used stand alone or it is in another nested typed DataGen, it will cause some tricky issues if we don't notice it when adding new IT cases.

Tests
TEST_PARALLEL=1 ./integration_tests/run_pyspark_from_build.sh -k 'cmp
Before: 430 passed, 19641 deselected, 1 warning in 201.35s (0:03:21)
After: 430 passed, 19641 deselected, 1 warning in 150.14s (0:02:30)

Closing it because it is in low priority, will work on it later and reopen it when it is ready.


[WIP] Fetch CSV data for previews from public url

When we switch to storing all attachments exclusively in Asset Manager, the CsvPreview class will no longer be able to read CSV data from the local NFS mount to generate the previews presented to the
user[1].
Instead we can generate the CSV previews by fetching the CSV file served at the public host. Currently that will in effect by reading the file from the NFS mount, but when we do change over to serving from Asset Manager it will fetch the CSV file from there.
Some CSV file attachments are almost 200Mb[2] so it is not practical to fetch the entire file as part of the preview request. Fortunately the preview functionality is configured to present at most 1,000 rows
and 50 columns to the user.
I've configured the Range request to request the first 30,000 bytes of the file. This is somewhat arbitrary at the moment, and I could do some work to set this to a limit that ensures 1000 lines of every currently uploaded csv attachment is fetched.
[1] For example: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/675936/Q4_2017_csv.csv/preview
[2]
https://www.gov.uk//government/uploads/system/uploads/attachment_data/file/399850/mappings-2015-01-27T10_26_13_00_00.csv/preview

@chrisroos the CSV preview isn't available in admin, only to the public. For example, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/675936/Q4_2017_csv.csv/preview. I'm hopeful that caching will mean we don't make too many requests.

@chrisroos the CSV preview isn't available in admin, only to the public. For example, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/675936/Q4_2017_csv.csv/preview. I'm hopeful that caching will mean we don't make too many requests.

Ah, OK. So to make sure I understand: a request for a CSV preview page will hit Whitehall, which will make a request to Asset Manager for a subset of the CSV file before sending it back to the client. The page sent back to the client will be cached by Fastly which will reduce the number of requests we're making to Asset Manager. Is that correct?

@chrisroos - it makes a request to the public URL for the asset which at the moment will hit Whitehall and fetch the file from NFS, but when we change over will hit Asset Manager. Does that make sense?

@chrisroos - it makes a request to the public URL for the asset which at the moment will hit Whitehall and fetch the file from NFS, but when we change over will hit Asset Manager. Does that make sense?

Yup, completely.

I've verified that this works on integration (I had to add a1ac03a to use basic auth in that environment). I've also set a value for MAXIMUM_RANGE_BYTES based on CSV files historically added to Whitehall. There's a small edge case remaining explained in the final commit, but I hope we can live with it to allow us to move on with migrating the assets over and come back to it shortly if we think it's necessary.
@chrisroos - could you review when you get chance?

Thanks @chrisroos - as discussed, I'm going to rebase this on master and merge.


progress tracking, tables, relationships, and API

Mechanisms to assign and track processing of audio segments are essential:
1. for users and researchers to know what data has already been reviewed
2. to assign audio clips for sampling in a strategic manner
3. to allow for large-scale visualization of data
4. to test for consistency/accuracy across users for species identification
5. Allow participants to identify easily when target species have been found (and for verification)
THIS ISSUE: progress tracking, tables, relationships, and API

include count of times a segment has been listened to? see below issue
include: number of hours spent listening on user profile
include: number of hours listened in site wide statistics

Related Issues

baw-client: progress tracking, using API [https://github.com/QutBioacoustics/baw-client/issues/263]
baw-client: progress tracking, visualization
Ecoacoustics: Long term goal: work assignment
Ecoacoustics: Long term goal: sampling work assignment

@atruskie if you could add more technical detail to this and related issues that would be most helpful.

Feedback added.


Issue#369 add new end user ids in experience events

Please link to the issue #369
Added the standard phone number and email address IDs to EndUser IDs. In the Visitor ID service they correspond to -
Namespace
|Title | id | code |
|Email | 6 | Email |
|Phone | 7 | Phone|

@kstreeter  - please put this in the June 15th milestone.

Why is this Adobe-centric stuff in /context?  Adobe-centric stuff goes in extensions.

@lrosenthol, we have been including all of the identity schemes in use directly in context/Identity, but these are all namespaced appropriately as Adobe extensions.  The rationale for this is to have the commonly used identity schemes documented all in one place.


Update bindings to LLVM 3.8

LLVM 3.8.0 was released and it would be nice if the bindings could be updated to the current version

Before that happens, I'd like to call attention to this issue because fixing it is a breaking change to the public API. If we decide to fix it, it might be best to do it now that a new version is coming.

Yes, that's a bummer. Ok, we'll plan to get this to 3.8 in the next few days.


Suggestion: can this be added to PyPi?

Thanks for the kind suggestion!
It is indeed on our plan but may take a long time for some internal review process. We will figure out other alternative ways first to ease the installation.


Paging

I would like to allow for paging.
For many of my data sources there is a lot of data. Some tables have a million row or more. I would not like to download a million rows for this component to page through. I would like to intercept API calls that need more data and translate that request into a request for just a page worth of data. So even though I have a million rows, I never transfer more than a page worth of data over the network at one time.

Can you see if this example solves your problem? https://github.com/IvanJosipovic/BlazorTable/blob/master/src/BlazorTable.Sample.Shared/Pages/ServerSideData.razor

I haven’t digested it yet, but this seems to do the trick. Thank you. The
key seems to be the ‘Records’ property. This has to be a like a class that
contains something like a count, page number, and page size. Correct?
On Thu, Mar 4, 2021 at 5:36 PM Ivan Josipovic<EMAIL_ADDRESS>wrote:

Can you see if this example solves your problem?
https://github.com/IvanJosipovic/BlazorTable/blob/master/src/BlazorTable.Sample.Shared/Pages/ServerSideData.razor
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/IvanJosipovic/BlazorTable/issues/267#issuecomment-791030222,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUFM3VGYHGIDTCDKBJAXHDTCAKQTANCNFSM4YUFYDRA
.

Sorry about that. I looked further in the code and could see a custom data
loader. Exactly what I was looking for. Thank you.
On Thu, Mar 4, 2021 at 6:49 PM Kevin Burton<EMAIL_ADDRESS>wrote:

I haven’t digested it yet, but this seems to do the trick. Thank you. The
key seems to be the ‘Records’ property. This has to be a like a class that
contains something like a count, page number, and page size. Correct?
On Thu, Mar 4, 2021 at 5:36 PM Ivan Josipovic<EMAIL_ADDRESS>wrote:

Can you see if this example solves your problem?
https://github.com/IvanJosipovic/BlazorTable/blob/master/src/BlazorTable.Sample.Shared/Pages/ServerSideData.razor
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/IvanJosipovic/BlazorTable/issues/267#issuecomment-791030222,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUFM3VGYHGIDTCDKBJAXHDTCAKQTANCNFSM4YUFYDRA
.

Just a clarification: The comment indicates that it is server side data. And the using statements pull in server side components. But I don't see any specific server side calls. It all seems to make HttpClient requests to get the data?


Add Travis CI config

The integration has been added to the repository. All that's left now is to set up the config file for our tests.

This is failing due to the .shopify-build directory no longer being a thing. We can disregard:

I just deleted the Buildkite pipeline


Strongly suggest rename

If you have serious ambitions for the project, and for the sake of potential contributors, I urge maintainers to consider a slightly different name. You don't want to attract the wrong kind of attention...

That is actually already planned: Pup Cup is coming 🙂

Indeed, don't worry there's no legal issue for having Mario Kart in the repository name, Pup Cup is still coming

Hey @Lunakepio, just a heads up, let's keep our repo going strong and trouble-free for as long as we can!

Will Pup Cup be multiplayer like Smash Karts?


Add a check_operation_status script

The current script can take a while for the statement to get anchored. Rather than continually run the curl, a check_operation_status.py script allows the user to run and wait for it to complete.

See https://github.com/datatrails/datatrails-docs/pull/726 for doc updates


is this library api based, if i want to use PHP instead of GO?

is this library api based, if i want to use PHP instead of GO?

Hi @rajjanorkar, sorry for late reply.
Shiori does have a REST API that you get after running shiori serve. There are several endpoints available that you can see here.

Thank you for your reply. I will investigate it further to see how i can do it.

Since it has been inactive for a while, I will close this issue for now. Feels free to reopen it if needed.
Thank you.

Hi @RadhiFadlillah
i have created this repository Bookmark-Manager for web application.
I have taken your vuejs component and plugged it into laravel backend app.
Thank you


Are you interested in getting PRs with more becnmarks?

I've adapted this to do Popcnt and various other things...

yes I definitely am :)

OK, i will try to push some popcnt/bmi2 stuff later tonight


EF Core TPC abstract classes unable to query implementations

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug
I've been attempting to get the table-per-concrete-type feature that was introduced in EF Core 7.0 to work with unions in Hot Chocolate and have no luck. Is this something that is supported via HotChocolate out of the box or do I need to supply additional configuration to get this to work with abstract classes?
I've got a basic schema that has a abstract type of FileSystemNode and then 2 descendants (FileSystemDirectory & FileSystemFile), I want to be able to query FileSystemNode and use unions to extract the additional information in the descendants, although none of the union types are available, only those members within FileSystemNode. I've watched the video posted to YouTube How to use GraphQL input unions with Hot Chocolate? and the setup is pretty much the same with the exception I'm using an abstract class as required by TPC instead of an interface.
Steps to reproduce
Here is my current setup as it stands as I assumed that the decedents would be picked up automatically without a additional configuration, if I am missing something please let me know :slightly_smiling_face:!
Note: using projection in this query causes the now closed issue outlined here to occur https://github.com/ChilliCream/hotchocolate/issues/1658 unsure if this should be reopped or if I'm doing something incorrect?
[ExtendObjectType(OperationTypeNames.Query)]
public class FindManyFileSystemNodes
{
    [UsePaging]
    // [UseProjection] https://github.com/ChilliCream/hotchocolate/issues/1658
    [UseFiltering]
    [UseSorting]
    public IQueryable<FileSystemNode> FileSystemNodes([ScopedService] ApplicationDbContext database)
    {
        return database.FileSystemNodes;
    }
}

public abstract class FileSystemNode : ITimestampableEntity
{
    ...
}

public class FileSystemFile : FileSystemNode
{
    ...
}

public class FileSystemDirectory : FileSystemNode
{
    ...
}

Relevant log output
No response
Additional Context?
No response
Product
Hot Chocolate
Version
13.0.0-preview.90

Do you mark the type as a union or interface ?

I've attempted to mark the FileSystemNode entity as a UnionType but this creates the following error The name 'FileSystemNode' was already registered by another type.. I'm also using HotChocolate.Types.Analyzers unsure if that's registering something it shouldn't, but it looks fine at a glance.
I've had no issue in the past when using interface types, and in Michael's video explaining HotChocolate's union types interfaces are used there too. I'm lead to believe it's due to an abstract class used instead but this is required for EF's TPC feature.
I've marked the union type as below as well as the descendants as follows, the query itself remains the same as referenced in the original issue description.
public class FileSystemNodeType : UnionType<FileSystemNode>
{
    protected override void Configure(IUnionTypeDescriptor descriptor)
    {
        descriptor.Type<FileSystemFileType>();
        descriptor.Type<FileSystemDirectoryType>();
    }
}

public class FileSystemDirectoryType : ObjectType<FileSystemDirectory>
{
    protected override void Configure(IObjectTypeDescriptor<FileSystemDirectory> descriptor)
    {
        descriptor
            .Field(p => p.Nodes)
            .UsePaging()
            .UseFiltering()
            .UseSorting();
    }
}

public class FileSystemFileType : ObjectType<FileSystemFile>
{
    protected override void Configure(IObjectTypeDescriptor<FileSystemFile> descriptor)
    {
    }
}

@PascalSenn pinning the type yields the following exception;
HotChocolate.SchemaException: For more details look at the `Errors` property.

1. For more details look at the `Errors` property.

1. Unable to infer the element type from the current resolver. This often happens if the resolver is not an iterable type like IEnumerable, IQueryable, IList etc. Ensure that you either explicitly specify the element type or that the return type of your resolver is an iterable type.
 (HotChocolate.Types.ObjectTypeExtension<Giantnodes.Service.Dashboard.Api.Resolvers.Explorer.FindManyFileSystemNodes>)

   at HotChocolate.Configuration.TypeInitializer.DiscoverTypes()
   at HotChocolate.Configuration.TypeInitializer.Initialize()
   at HotChocolate.SchemaBuilder.Setup.InitializeTypes(SchemaBuilder builder, IDescriptorContext context, IReadOnlyList`1 types)
   at HotChocolate.SchemaBuilder.Setup.Create(SchemaBuilder builder, LazySchema lazySchema, IDescriptorContext context)
   at HotChocolate.SchemaBuilder.Create(IDescriptorContext context)
   at HotChocolate.SchemaBuilder.HotChocolate.ISchemaBuilder.Create(IDescriptorContext context)
   at HotChocolate.Execution.RequestExecutorResolver.CreateSchemaAsync(String schemaName, RequestExecutorSetup options, RequestExecutorOptions executorOptions, IServiceProvider serviceProvider, TypeModuleChangeMonitor typeModuleChangeMonitor, CancellationToken cancellationToken)
   at HotChocolate.Execution.RequestExecutorResolver.CreateSchemaServicesAsync(String schemaName, RequestExecutorSetup options, CancellationToken cancellationToken)
   at HotChocolate.Execution.RequestExecutorResolver.GetRequestExecutorNoLockAsync(String schemaName, CancellationToken cancellationToken)
   at HotChocolate.Execution.RequestExecutorResolver.GetRequestExecutorAsync(String schemaName, CancellationToken cancellationToken)
   at HotChocolate.Execution.RequestExecutorProxy.GetRequestExecutorAsync(CancellationToken cancellationToken)
   at HotChocolate.AspNetCore.HttpPostMiddlewareBase.HandleRequestAsync(HttpContext context)
   at HotChocolate.AspNetCore.HttpPostMiddlewareBase.InvokeAsync(HttpContext context)
   at Microsoft.AspNetCore.Builder.EndpointRouteBuilderExtensions.<>c__DisplayClass20_0.<<UseCancellation>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Watch.BrowserRefresh.BrowserRefreshMiddleware.InvokeAsync(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

@PHILLIPS71  ok, can you provide a simple repro of the bug?

@PascalSenn I've created a repo here https://github.com/PHILLIPS71/hc-5643, let me know if there is any other information required 🙂

@PascalSenn is there any plans to resolve the issue in any upcoming release? Or are you able to point me in the right direction on where to start looking if I were to attempt to resolve the issue myself, or at least provide more details in this issue?

I've done some more digging and it looks as though in v13.4.0 it's partially working, if you pin the GraphQL type it's able to resolve the query and return some data, however, if you don't pin the type in the query you will get a HotChocolate.SchemaException.
[GraphQLType<ListType<FileSystemNodeType>>]
[UsePaging]
[UseProjection]
[UseFiltering]
[UseSorting]
public IQueryable<FileSystemNode> FileSystemEntries([Service] ApplicationDbContext database)
{
    return database.FileSystemNodes;
}

It's also worth noting that while the above does work if you pin the type, it does not handle owned entity types and always produces a null value for those objects, it looks as though its not selecting those values when constructing the sql query.
It doesn't work if the union type is part of a nested collection, for example if I had a library that has a collection of FileSystemNode objects, the collection that is returned is always an empty array. I've attempted to pin the type on the nested collection but it's produces the same result, it also looks as though the nodes are not being selected via the sql query.
public class LibraryObjectType : ObjectType<Library>
{
    protected override void Configure(IObjectTypeDescriptor<Library> descriptor)
    {
        descriptor
            .Field(p => p.Nodes)
            .Type<ListType<FileSystemNode>>()
            .UseProjection()
            .UseFiltering()
            .UseSorting();
    }
}

I've done some more digging and it looks as though in v13.4.0 it's partially working, if you pin the GraphQL type it's able to resolve the query and return some data, however, if you don't pin the type in the query you will get a HotChocolate.SchemaException.
[GraphQLType<ListType<FileSystemNodeType>>]
[UsePaging]
[UseProjection]
[UseFiltering]
[UseSorting]
public IQueryable<FileSystemNode> FileSystemEntries([Service] ApplicationDbContext database)
{
    return database.FileSystemNodes;
}

It's also worth noting that while the above does work if you pin the type, it does not handle owned entity types and always produces a null value for those objects, it looks as though its not selecting those values when constructing the sql query.
It doesn't work if the union type is part of a nested collection, for example if I had a library that has a collection of FileSystemNode objects, the collection that is returned is always an empty array. I've attempted to pin the type on the nested collection but it's produces the same result, it also looks as though the nodes are not being selected via the sql query.
public class LibraryObjectType : ObjectType<Library>
{
    protected override void Configure(IObjectTypeDescriptor<Library> descriptor)
    {
        descriptor
            .Field(p => p.Nodes)
            .Type<ListType<FileSystemNode>>()
            .UseProjection()
            .UseFiltering()
            .UseSorting();
    }
}

closing as it's an issue with ef core and owned types using TPC see https://github.com/ChilliCream/graphql-platform/issues/6423


add Series.abs/1

https://pola-rs.github.io/polars/py-polars/html/reference/series/api/polars.Series.abs.html

Thank you! :purple_heart:


Some evaluated locations are not found

It can sometimes happen that something that was evaluated is not found in the traceCorrel. We should investigate.

Sadly, #95 doesn't quite cut it: we don't actually get precedence information until after renaming, so we can't find locations that are "parenthesized" by precedence.


Further explanation of 'error: unequal lengths in sequence and overlap file for sequence'

Hi,
Attempting to implement racon in nanopore polishing pipeline.
Using bowtie2 for alignment of concatenated fastq illumina read files, passing this and the  concatenated read file to racon.
I get the following error code during execution:
Begining racon polish iteration: 1...
[racon::Polisher::initialize] loaded target sequences
[racon::Polisher::initialize] loaded sequences
[racon::overlap::find_breaking_points] error: unequal lengths in sequence and overlap file for sequence M03542:170:000000000-BL6WR:1:1101:22727:1756!
Could you elaborate on the error message so I can troubleshoot?
Greatly appreciate you time!
Cheers,
Ben
P.S. Here is code from pipeline:
RACONBASEDIR=racoon0"$I"
mkdir $RACONBASEDIR
INDX="$RACONBASEDIR"/"$GENOMEBASE""$I"
printf "Building bowtie2 index: $INDX\n"
printf "Buidling...\n"
sleep 1
bowtie2-build $ITERGENOME $INDX
printf "Completed buidling bowtie2 index.\n\n"

#	Align Illmuina fastq reads
printf "Concatenating Illumina reads for alignment...\n"
CATREADS="$RACONBASEDIR"/temp.fastq.gz
zcat $ILLREAD1 $ILLREAD2 | gzip -c > $CATREADS
printf "Completed concatenation.\n\n"

ALIGNDREADS="$RACONBASEDIR"/"$GENOMEBASE"."$I".sam
printf "Aligning $CATREADS to $INDX with bowtie2...\n"

bowtie2 -p 12 --fast -x $INDX -U $CATREADS -S $ALIGNDREADS
printf	"Completer bowtie2 alignment.\n"
printf "Aligned reads ouput file: $ALIGNDREADS\n\n"

#	Pass genome.iteration, illumina.reads, and iteration.alignment to racon
printf "Begining racon polish iteration: $I...\n"
POLISHEDGENOME="$RACONBASEDIR"/"$GENOMEBASE".polish."$I".fasta
racon $CATREADS $ALIGNDREADS $ITERGENOME > $POLISHEDGENOME

Hello Ben,
I think that your paired-end reads have equal names up to the first white space. Please verify that.
Best regards,
Robert

Hi Robert,
Yes read 1 and read 2 for each paired set have equal read names up to the first white space. Is this the discrepancy with racon? Different lengths for each of the paired reads that have identical read names.
Is there a more appropriate way to handle paired-end reads using racon?
Cheers,
Ben
$ zcat 24_S24_L001_R1_001.fastq.gz |  grep -e @M | head
@M03542:170:000000000-BL6WR:1:1101:22603:1684 1:N:0:ATCTCATG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:15599:1718 1:N:0:ATCTCAGG+TTTCCTCT
@M03542:170:000000000-BL6WR:1:1101:14294:1748 1:N:0:ATCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:21219:1750 1:N:0:ATCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:22727:1756 1:N:0:ATCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:16018:1761 1:N:0:CTCTCAGG+TATCCTTT
@M03542:170:000000000-BL6WR:1:1101:8108:1784 1:N:0:CTCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:19095:1785 1:N:0:CTCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:19081:1806 1:N:0:ATCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:15653:1812 1:N:0:ATCTCAGT+TATCCTCT
$ zcat 24_S24_L001_R2_001.fastq.gz |  grep -e @M | head
@M03542:170:000000000-BL6WR:1:1101:22603:1684 2:N:0:ATCTCATG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:15599:1718 2:N:0:ATCTCAGG+TTTCCTCT
@M03542:170:000000000-BL6WR:1:1101:14294:1748 2:N:0:ATCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:21219:1750 2:N:0:ATCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:22727:1756 2:N:0:ATCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:16018:1761 2:N:0:CTCTCAGG+TATCCTTT
@M03542:170:000000000-BL6WR:1:1101:8108:1784 2:N:0:CTCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:19095:1785 2:N:0:CTCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:19081:1806 2:N:0:ATCTCAGG+TATCCTCT
@M03542:170:000000000-BL6WR:1:1101:15653:1812 2:N:0:ATCTCAGT+TATCCTCT

Hi Ben,
racon treats all reads as single ends so no two reads should have the same identifier up to the first white space. You can add 1 and 2 before the white space or remove all white spaces from read names. Don't forget to update your overlap file or recreate it.
Best regards,
Robert

I have a similar issue: how do I deal with multi-mapping reads?

Hello,
could you please clarify what are multi-mapping reads? Do you have the same case where Illumina reads from a pair have equal names up to the first white space? Or something else?
Best regards,
Robert

For instance the read 'HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547'
this one is twice in the read fastq because they are paired end:
$ zgrep -A 3 "HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547" reads.fastq.gz
@HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547
CCGCTGATCACCGCGCAAACACACAGCAGCAAAATTTCCTCCAGATCATAAAGCTTGTTCTTGTCCGAGCGGGGATCTTTGATCGTCGAAAAATGCGAGA
+
CCCFFFFFHHFHHJJJIJJJGIJJJJJJJJIJIGIJJJJIJIJJIIIIIJHIJJHGHEHFFFFFFEDDBDDDDDDBDDDDDDCDD<89>BBB<ACDDDD#
--
@HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547
TCGATATGATAGTCGTGGCCCGCCTTCTTGTTGAGCGCGATGGTAGCGCCGCCGTGCTTGGGTTGTTTGGTCTTTGCCATCGTGGAAGTGTAACCCAAGA
+
CCCFFFFFHHHHHJJJJJJJJJIJIJJIIIJJIFGIIIJGIJJGIJJHGFACDDBDDDDDDD?@DBBDCDACDDD@CCDCDDBBD?CDCCDDEDDDDDDB

and they are therefore 4 times in the alignment file:
$ grep HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547 mapped_ill-pe_hybr-genome.bwamem.sam
HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547	0	NODE_1_length_96688_cov_60.6554	1	60	5S95M	*	0	0	    CCGCTGATCACCGCGCAAACACACAGCAGCAAAATTTCCTCCAGATCATAAAGCTTGTTCTTGTCCGAGCGGGGATCTTTGATCGTCGAAAAATGCGAGCCCFFFFFHHFHHJJJIJJJGIJJJJJJJJIJIGIJJJJIJIJJIIIIIJHIJJHGHEHFFFFFFEDDBDDDDDDBDDDDDDCDD<89>BBB<ACDDDD#	NM:i:0	MD:Z:95	AS:i:95	XS:i:60
HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547	0	NODE_15_length_49256_cov_64.9048	49063	60	100M	*	0	0	TCGATATGATAGTCGTGGCCCGCCTTCTTGTTGAGCGCGATGGTAGCGCCGCCGTGCTTGGGTTGTTTGGTCTTTGCCATCGTGGAAGTGTAACCCAAGA	CCCFFFFFHHHHHJJJJJJJJJIJIJJIIIJJIFGIIIJGIJJGIJJHGFACDDBDDDDDDD?@DBBDCDACDDD@CCDCDDBBD?CDCCDDEDDDDDDB	NM:i:0	MD:Z:100	AS:i:100	XS:i:0

@sarah872, racon treats all reads as single ends so no two reads should have the same identifier up to the first white space. You should add 1 and 2 at the end of each read in a pair (e.g. @HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547:1 and @HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547:2). Don't forget to recreate the SAM file!

Thanks. Do you know of any available script that executes that for me?

Very nice of you!
unfortunately I am getting the following error:
  File "rename_pe_reads.py", line 64, in <module>
    read_set = sets.set()
AttributeError: module 'sets' has no attribute 'set'

It should be sets.Set() as above.

still the same error:
AttributeError: module 'sets' has no attribute 'Set'

Are you using python3.x? If so replace the line with read_set = set().

the script is running, but it is running without finishing. Also the file size increases stadily, from the original 1.7 GB file (7 Mio reads) to 34GB (although I stopped the script then...)

Ugh my bad, I updated the above script with everything so far (I forgot to clear variables between reads :D).

perfect, works!
thanks!

Hi all,
Would it not be enough just to consider only R1 reads?
Thank you in advance,
Michal

If you have decent coverage, sure.
Best regards,
Robert

I have two types of FASTQ files:
This one is a new one with sanger style quality score:
> zcat 1740D-43-06_S0_L001_R1_001.fastq.gz | head -n 1
@E00526:39:HNMN5CCXY:6:1101:3792:993 1:N:0:NTTCAGAA+NTTCGCCT
> zcat 1740D-43-06_S0_L001_R2_001.fastq.gz | head -n 1
@E00526:39:HNMN5CCXY:6:1101:3792:993 2:N:0:NTTCAGAA+NTTCGCCT

the below dataset (old style Phred score Illumina 1.3+) is for a different organism:
> zcat OZBenth6_R1.fq.gz | head -n 1
@HWI-ST945_0069:6:1101:1694:1991#NNNNNN/1
> zcat OZBenth6_R2.fq.gz | head -n 1
@HWI-ST945_0069:6:1101:1694:1991#NNNNNN/2

Will you script be able to handle both types of headers?
Thank you in advance.
MIchal

I think it should.
Best regards,
Robert

First dataset example:
> zcat 1740D-43-06_S0_L001_R1_001.fastq.gz | head -n 1
@E00526:39:HNMN5CCXY:6:1101:3792:993 1:N:0:NTTCAGAA+NTTCGCCT
> zcat 1740D-43-06_S0_L001_R2_001.fastq.gz | head -n 1
@E00526:39:HNMN5CCXY:6:1101:3792:993 2:N:0:NTTCAGAA+NTTCGCCT

You script generated this:
@E00526:39:HNMN5CCXY:6:1101:3792:9931
NTCATAACCATAGAATATGTAAATCTCTAAGTAGTAGTCTAAGACACCATATCCCTTTCGGGATCCGCCTATTCATTTATGCCCTACTACTTGAATACTTCTAATTATATTCCATTACTTTGCCTATCCAGTTCCGCTTGCATCGATATTT
+
#A<AFFFJJJ7FF7-A-J<FA-AF<FFFJA<JAF-A7-<-JFJJJ<<<JJJ--F7F<AJF-FFJFFFFF-J---7----<----A--77JFJA-----77-7---A----7----7--7--7----7---7---------7--7-------
@E00526:39:HNMN5CCXY:6:1101:3792:9932
NTGGCGCTATTATTATCTCTAAGGGAAGTTAAGGATGAACTAGAAGCAGATAGATTGGTTAGACATGGTTGAACCGGCATAGCGATGGCGATGCATGCGGGGCTCGATGGCCGTCGTAATTGATTCTATTTTGAAGTATTCAAGTATTAGG
+
#<<A<A---FAFA<-FA7FFJ7AJFJ<F-FF-F<<-F-<FAF<-FJ7--<A-<-7<FJ-7-F-A-<AAFJJJ<-JFA--FF<-77AFF7<-AFJAF<F7AAA--7-7A-F-AF77A7-7-<<-7F--7<<J-A-7-AAJF7A<-<-7<A-<

Second dataset example
> zcat OZBenth6_R1.fq.gz | head -n 1
@HWI-ST945_0069:6:1101:1694:1991#NNNNNN/1
> zcat OZBenth6_R2.fq.gz | head -n 1
@HWI-ST945_0069:6:1101:1694:1991#NNNNNN/2

You script generated this:
@HWI-ST945_0069:2:1101:1483:1976#GNNNNN/11
NAGGCTCATTCTTAAAACCTATGCCATGCTTGGTTGACTTCAAATTAACAAGGACCTCATTCAANNTNTTGNNNNNAGANTTAAACTNNNNNAAGGTACC
+
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
@HWI-ST945_0069:2:1101:1483:1976#GNNNNN/21
NCANATTGCCCAAATATCAAAAAGGAGATGAAACCTTTTAAGAAGCAACGTGCATACATTTCTTGGGGAGGAGACTTCGGAGATGAGTCAAGCGAAGAAG
+
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

Is it correct?
Thank you in advance,
Michal

Looks fine :)
Best regards,
Robert

@sarah872, racon treats all reads as single ends so no two reads should have the same identifier up to the first white space. You should add 1 and 2 at the end of each read in a pair (e.g. @HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547:1 and @HWI-ST863:386:C5Y8UACXX:3:1107:14282:8547:2). Don't forget to recreate the SAM file!

Hi, for my help, what do you mean 'Don't forget to recreate the SAM file' .?
Thank you

If you have created the SAM file from a FASTQ file which has sequences with identical names up to the first white space, and then change the header to distinguish them, you have to update the existing SAM file with new sequence headers or rerun the chosen tool to obtain a new SAM file.
Best regards,
Robert

Thanks Rvaser,
The scripts works perfectly.


fix(instrumentation-mysql2): sql-common should be a dependency

Which problem is this PR solving?

sql-common was accidentally listed as a dev dependency

Short description of the changes

Moves it to dependencies

Probably warrants a patch release given this makes the instrumentation unusable unless the pg instrumentation is also in the tree (it's also on me for putting it in the wrong place to begin with, woops)