fix: we weren't handling time formatting in any way

Fixes #1144

IIRC we intentionally weren't handling time when timeFormat was undefined. I think that's so it could work as both a date field and a datetime field.
Maybe @DirtyF or @jamespohalloran remembers?

I just thought of something I need to check. So I will look into it and reopen.

This has been a journey.

With my initial PR I hadn't considered how dates were actually being saved in the front-matter. They were being saved in the user-provided format, which might not be the worst but could end up painting the user into a corner if they ever decided they wanted to change formats.
I had to make some adjustments for timezones. I went ahead and made the assumption that for now we always want to work in UTC. The best assumption we can make right now is that when a user inputs 8:00AM they want everyone else to also see 8:00AM.
There was seemingly another bug where after saving the date would get inserted into the frontmatter as 2020-08-02T12:12:12Z instead of '2020-08-02T12:12:12Z', this was causing a parsing error. To fix this I made the parse function return a string in the default datetime format.

It seems to be all working now.

I'm still seeing some weird behavior.

I'll dig in a bit more, maybe give it a try in another project other than tinacms.org. Thanks!

I was able to reproduce the issue by not passing anything to dateFormat while also specifying a timeFormat. dateFormat.ts specifies a format to use in this case, but that format was not actually being passed into the ReactDateTime component.
Passing the default dateFormat to ReactDateTime seems to fix the issue and it will just get overwritten the same way timeFormat={false} is being overwritten.
It might be worth it to make this field type more flexible, because at the moment it can't be configured to act exclusively as a time-picker (though that doesn't seem like a very desirable field type, and this limitation is reflected in the docs).

@kendallstrautman @DirtyF  let me know if you can take another look at this!
I think there is still some funkiness that people can get themselves into if they provide weird formats (i.e., specifying timeFormats w/ incomplete dateFormats, specifying dateFormat=false) but I don't think it's worth it to fix those or even trying to document all the ways people could get themselves into trouble.


Issue with Select2 inside a pop-up

Hey, the plugin looks great on the website. I have used the plugin inside a pop-up which show/hide based on jQuery clicks on an anchor.
The issue is I need to close the pop-up when user clicks anywhere in the body except the pop-up.
I have written the below code to achieve the same.
jQuery(document).mouseup(function(e) {
	if(jQuery(e.target).closest('.callback-details').length == 0) {
		$('.callback-details').slideUp('500ms');
	 }
});

The structure of select2 generates outside of my pop-up during run time. Therefore the pop-up is getting closed even when the user clicks select2 box inside the pop-up.
Any suggestions for this issue?

Hi, is there any documentation on how to use the AttachContainer module? I've included the full version of select2, but i can't get this working. I'm having the select2 elements inside a popup(magnific-popup), but due to some reasons the search input can not be focused. It will work when using dropdownParentto attach the dropdown somewhere else, but then the position css of the dropdown is incorrect.

Here's a jsfiddle that shows how to decorate the dropdownAdapter with the AttachContainer decorator.
http://jsbin.com/yisetuhuji/2/edit?html,js,output
I've also included the DropdownSearch decorator in there considering how commonly used it is.

Thanks @kevin-brown, this is super helpful. Follow up question: Is there a way to implement your example without the DropdownSearch decorator? When I try to do it, I get the following errors on the console:
Uncaught TypeError: undefined is not a function select2.full.js:4855
S2.define.Select2._bindAdapters select2.full.js:4748
S2.define.Select2 select2.full.js:5794 (anonymous function)

@kevin-brown Thank you very much for sharing this!

In 4.0, you can use the AttachContainer module to attach the dropdown to the parent container, but keep in mind that this prevents things like dropping up

@kevin-brown, is this still the case? And if so, how hard it will be to fix this?


Quick fix in setup.py

Hi guys, there is an issue preventing the installation of CommPy 0.7.0 via PIP in Windows. When attempting to do pip install scikit-commpy on Windows, version 0.6.0 is installed instead, due to a bug in setup.py in version 0.7.0. I don't know if the same issue appears in other OS.  Looking at the error logs and similar issues reported on StackOverflow, I found a quick fix.
In setup.py, change:
LONG_DESCRIPTION = open('README.md').read()

to
LONG_DESCRIPTION = open('README.md', encoding="utf8").read()

That should fix the pip installation on Windows. I tested it and this fix allowed me to install version 0.7.0 on my Windows PC.

Thanks :+1:


feat: create sync command

Description
created a command to synchronize ReChunk keys with native files

yeah, I think it's a great idea, and that way it'll be easier to update the files

One thing I think we should think about is utilizing the parsers for plist and xmls i.e. withInfoPlist and withStringsXml - what's your thought on this?

yeah, I think it's a great idea, and that way it'll be easier to update the files

One thing I think we should think about is utilizing the parsers for plist and xmls i.e. withInfoPlist and withStringsXml - what's your thought on this?

this fast-xml-parser for android
and plist for ios
correct?

yeah, I think it's a great idea, and that way it'll be easier to update the files

One thing I think we should think about is utilizing the parsers for plist and xmls i.e. withInfoPlist and withStringsXml - what's your thought on this?

this fast-xml-parser for android and plist for ios
correct?

yep those are the perfect libs for this!

yeah, I think it's a great idea, and that way it'll be easier to update the files

One thing I think we should think about is utilizing the parsers for plist and xmls i.e. withInfoPlist and withStringsXml - what's your thought on this?

this fast-xml-parser for android and plist for ios
correct?

yep those are the perfect libs for this!

I chose this package for plist parser - simple-plist because it has types for typescript


Fixed negative index exception

Related to this change:
https://github.com/divio/django-cms/commit/aa1e0ce2d3e54f6dceec6eda919dddec5da899b0#L0L120
Children variable was changed from empty list to queryset, but a negative index was still used later on causing an exception to be thrown.  Also changed how children length is calculated, opting to use the queryset method count instead of len function, to avoid loading the data before necessary.

Just noticed this is a duplicate of the following issue, and an identical solution was proposed there by not accepted.
https://github.com/divio/django-cms/issues/1099


The ConfigMap "rh-operators" is invalid while installing on upstream k8s

Following the instructions at Install the latest released version of OLM for upstream Kubernetes on k8s 1.11, I get the following message:
$ kubectl apply -f deploy/upstream/manifests/latest/
clusterrole.rbac.authorization.k8s.io/system:controller:operator-lifecycle-manager configured
serviceaccount/olm-operator-serviceaccount unchanged
clusterrolebinding.rbac.authorization.k8s.io/olm-operator-binding-kube-system configured
customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com configured
catalogsource.operators.coreos.com/rh-operators unchanged
deployment.apps/olm-operator unchanged
deployment.apps/catalog-operator unchanged
clusterrole.rbac.authorization.k8s.io/aggregate-olm-edit configured
clusterrole.rbac.authorization.k8s.io/aggregate-olm-view configured
apiservice.apiregistration.k8s.io/v1alpha1.packages.apps.redhat.com configured
clusterrolebinding.rbac.authorization.k8s.io/packagemanifest:system:auth-delegator configured
rolebinding.rbac.authorization.k8s.io/packagemanifest-auth-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/packagemanifest-view configured
clusterrolebinding.rbac.authorization.k8s.io/package-apiserver-clusterrolebinding configured
secret/package-server-certs unchanged
deployment.apps/package-server unchanged
service/package-server unchanged
The ConfigMap "rh-operators" is invalid: metadata.annotations: Too long: must have at most 262144 characters

Is there a workaround for annotations length limit?

@ecordell thanks! that worked.
Also, I was following the Getting Started Guide's Deploy the Operator section and tried to install memcachedoperator csv after deploying OLM, but I get this error.
$ kubectl apply -f memcachedoperator.0.0.1.csv.yaml
error: unable to recognize "memcachedoperator.0.0.1.csv.yaml": no matches for kind "ClusterServiceVersion-v1" in version "app.coreos.com/v1alpha1"

I'm not aware of much of the internals of OLM at the moment. Some hint would be great.

@darkowlzz Great!
That example is unfortunately a little out of date - but I'm betting that if you just remove the -v1 prefix from the example types, it will still work. (i.e. ClusterServiceVersion-v1 -> ClusterServiceVersion)
Closing since your original issue is fixed - please open an issue on the Getting Started repo if you have more problems with that.


docs: update the example to include raw queries

Describe this PR
Updating the documentation to also include raw queries as a reference being documented and how to use it with an existing seed.js/seed.ts file
Changes
A documentation change
What issue does this fix?
No specific issue opened for this.
Any other relevant information

@zachtil can you take a look at this docs one too?

Thanks for pushing. Let me know if there are any updates needed here.


KNN Weights unexpected behavior

Hi,
I have found some unexpected behavior in the KNN weights construction function in a large point dataset with potentially coincident points. I cannot share the data, but I can attempt to clear sharing a selection.
In this dataset, there's a failure at line 106 in the KNN.__init__ function of Distance.py when the row.remove(i) command is supposed to remove self-neighbors. This appears to occur when the KNN query does not return the self-neighbor, but instead returns a different point. This point appears to always be coincident.
Since i is not returned, the row.remove(i) call fails. The full traceback isn't quite informative:
------------------------------------------------------------------------
ValueError                             Traceback (most recent call last)
<ipython-input-41-2b82caf342d3> in <module>()
----> 1 dW.KNN(np.vstack(filtered.XY.values), k=5)

/home/ljw/Dropbox/dev/libpysal3/libpysal/weights/Distance.py in __init__(self, data, k, p, ids, radius, distance_metric)
    105             row = row.tolist()
    106             try:
--> 107                 row.remove(i)
    108             except ValueError:
    109                 print(row, i)
    110                 raise

ValueError: list.remove(x): x not in list

I think this happens when the focal point is coincident with more than k other points and the query does not include the focal point in the knn set.
We could do something like:
if i in row:
      row.remove(i)
else:
     warning about coincident points

Of course you could also have coincident points and i being in row, so  the check wouldn't be a way to catch all coincident cases.
Alternatively, we don't check for coincident cases but simply change
line 109
to:
neighbors[focal] = [j for j in row if j!=i]

and drop the row.remove(i)

I'll have to inspect to figure out if that's the condition here. I know it's a dataset with many coincident points, so I'm not sure if there are that many causing this to occur.
If we don't end up removing at least one of the points, won't that row have too many neighbors? We do a k+1 remove 1, so if j != i always, wont len(neighbors[focal]) be one too long?

Good catch on the k+1. We do have to remove one for each focal unit, and that was based on the assumption that the focal unit would be among the query results.

This should've been filed in libpysal! my bad.
Regardless, this does appear to be an issue when there's just way too many coincident points. The data I'm working with is parcel delivery data, so some buildings have many units which each can have multiple parcels in a day. On a KNN of 20, you can get some real bad coincident point issues:

I'll draft a fix for my case and sub to libpysal.

I guess that'd mean it's impossible to to enforce k-nearest non coincident points, though. hmm...

related back to #579

For note, I never managed a fix for this. I'm opening a tracking issue on libpysal before I lose this thread again.

Any idea if this will be fixed soon?

This stalled because we haven't quite decided what "fixed" looks like. Is it "fixing" to not match coincident points? or, easier, "fixing" to just avoid self-matching?
This isn't on the top of anyone's queue at the moment, and a PR to fix the second sense would be simple: remove x if x is in the list, and otherwise, remove one arbitrarily.
Contributions welcome!


rrf/change fires AFTER custom change handler

The Problem
rrf/change action fires AFTER custom onChange handler. So the outdated data from the store is being passed to it.
Steps to Reproduce
For example, I have a <Control.Checkbox/> component in my app.
<Checkbox
   model="application.sms_notification"
   label="SMS notification"
   onChange={this.handleCalculatorChange} / *custom handler which collects data
from application store and sends API request with it immediately */
 />

What happens when the checkbox is checked/unchecked?
Expected Behavior
First rrf/change fires and updates the store, than the custom handler is executed.
Actual Behavior
rrf/change action fires AFTER handleCalculatorChange. So the outdated data is being passed!
Reproducible Code Example
https://codesandbox.io/s/nr8o8rxmkp

I just ran into this same issue.

Any updates on this?

Same here. It is a headache.


Project proposal applying to move ACA-Py to OWF

This PR contains a project proposal applying to move the Hyperledger Aries sub-project Aries Cloud Agent Python (ACA-Py) to the OpenWallet Foundation.  ACA-Py (Aries Cloud Agent Python) is an open-source framework designed to facilitate the creation, management, and utilization of decentralized digital identities. ACA-Py enables secure, interoperable communication between entities using decentralized identifiers (DIDs) and verifiable credentials, adhering to standards such as DIDComm, OpenID4VCs and the W3C Verifiable Credentials. This proposal has been vetted by the Aries community, and we look forward to continuing that discussion in the OWF.
@dbluhm, @tkuhrt, @jamshale, @TelegramSam, @jljordan42 @brianbehlendorf, @WadeBarnes, @andrewwhitehead, @chumbert
ACA-Py includes several repositories that would transition together with the core ACA-Py repo, as noted in the project proposal.
The Aries community will shortly follow up with 3 other project proposal PRs to apply to move other Aries sub-projects to OWF, and a couple of others could follow over time.  We also will be submitting a proposal to start a "Wallet Interoperability" Special Interest Group (SIG) at OWF that will be a forum for promoting interoperability across these and other related projects, specifications and standards.  This note will be updated with links to the related PRs for:

Askar
Wallet Test Tools -- to be grouped with the OWF's Mobile Wallet Test Harness
DIDComm Mediator Service
Wallet Interoperability Special Interest Group

@swcurran : Thank you for the proposal. I would like to add this as an agenda item for our next TAC meeting, which is October 18th @ 7:00 AM US/Pacific. Would that time work for the team to represent this proposal?

Sounds good.  I’ll let others in the ACA-Py community know.  Presumably, we’re cover all of the related proposals at the same time?

Not sure if we will make them through all of them, but we can definitely bring them to the attention of the TAC. I figure we start with this one and see if we can get approval for all of them, but we do tend to take a while. :)

October 18th @ 7:00 AM US/Pacific

Ugh...got the wrong date. This should have said September 18th @ 7:00 AM US/Pacific. My bad.

We support this project. Many members of GAN use ACA-Py heavily.


IllegalStateException in sdk version 23

i am running this project ins
when i'm using default configuration as follow:
'com.jakewharton:butterknife:6.1.0'
compileSdkVersion 22
buildToolsVersion "23.0.2"
targetSdkVersion 22
that work fine!
But when i change the sdk version to 23 as follwo:
compileSdkVersion 23
targetSdkVersion 23
then i can not start the app, with a exception
java.lang.IllegalStateException: Required view 'ivMenuUserProfilePhoto' with ID<PHONE_NUMBER> for field 'ivMenuUserProfilePhoto' was not found. If this view is optional add '@Optional' annotation.
those activities cause the exception are:
BaseActivity
 BaseDrawerActivity
MainActivity
it seems no matter the version of butterknife
i had try to use version 7.1.0
still useless
am i do something wrong??
thanks for helping!!

If you are using a navigation view, I've seen that issue as well.  For a workaround I removed the header layout from the Navigation view and the bind calls and then in onCreate I used inflateHeaderView and found the views based off that:
View headerLayout = mBaseDrawerActivityNavigationView.inflateHeaderView(R.layout.base_drawer_activity_left_drawer_header_layout);
mName = ButterKnife.findById(headerLayout, R.id.base_drawer_activity_left_drawer_name);
mProfileImage = ButterKnife.findById(headerLayout, R.id.base_drawer_activity_left_drawer_profile_image);

Yep, that view comes from the defined header view.
Dupe of #395. See the b.android.com link in that issue for more info.


Allow ddoc_cache to be fully disabled

Overview
Currently it is impossible to disable the ddoc_cache by specifying -ddoc_cache max_objects 0 in vm.args. This allows such a configuration, paired with a similar change in ets_lru.
Testing recommendations
Add -ddoc_cache max_objects 0 in vm.args and start up couch. It should run correctly.
GitHub issue number
Related to #559 but does not fix it.
Related Pull Requests
https://github.com/apache/couchdb-ets-lru/pull/4
Checklist

[X] Code is written and works correctly;
[ ] Changes are covered by tests;
[ ] Documentation reflects the changes;

+1


.ariaDisabled(true) not affecting tabindex

Is your feature request related to a problem?
Using .ariaDisabled(true) on an interactive element doesn't remove it from tab selection as it retains its tabindex='0' property.
Describe the solution you'd like.
Making disabled elements tabindex='-1' would be in line with the hover effect signifying they cannot be selected/interacted with.

Yeah.  It should probably be doing that.

This is resolved, so closing the issue is okay. 👍🏻
For future reference, however, I generally close resolved issues myself either automatically by commit message or, failing that, manually.
In this case the resolves #219 bit in the titles would have triggered auto-closing upon the develop branch being merged into master.


Where do you override the generate method while inference (BART)

Hello,
I saw the code in the text_step
            output = self.generate(
                input_ids=input_ids,
                vis_inputs=(vis_feats, vis_pos),
            )

THe original Huggingface BART's generate function didn't include your newly defined vis_inputs,
where do you overide it?
Thank you.

@zuujhyt Hi, I have the same question. Did you figure out how to override the generate method while inference?
Thanks.

I didn't change the generate method of GenerationMixin class. The generate method indeed supports to use extra parameters via **kwargs
Instead of changing generate, I updated the methods (prepare_inputs_for_generation, _expand_inputs_for_generation) that are used in generate that use **kwargs, as I did in modeling_t5.py and modeling_bart.py


URL tracing flag

To help understand what endpoints the tool is using under the hood, use --trace to show the URLs called.

Good suggestion. Done.

Thanks for your contribution.


Partial update deserialization

It would be very useful if deserialization could be used for partial updates. For example I have a struct and I want to deserialize into it but allow some fields to me missing.
Right now the only real option is to make every field Option<T>. This works but makes working with the structure very inconvinent. You will almost always want to create a second structure without all of the Option fields and that increases maintenance burden to keep the two in sync.
If you write your own deserializer I think you can accomplish this but again, there is a lot of maintenance burden for this.
I think one way to implement this would be add an option to the deserialize_in_place generation to serde_derive to not raise an error if fields are missing. This would allow just updating the relevant fields. This is safe because all of the fields were valid before deserialize_in_place was called so it is fine to keep them as they were.
Example use cases:

Configuration merging. This would enable efficiently and easily deep-merging multiple objects together.
Multiple default values. This is sort of the same as the previous but right now your only options for defaults are setting them in #[serde(default="some::path")] which hardcodes them. If you could merge on top of an existing object that existing object becomes the defaults.
Delta updates. This is useful for HTTP PATCH or other remote communication. It allows you just to send over the changed fields.

I would prefer not to build anything for this into the serde crate, but it sounds like something that could be implemented outside of serde, covering all of the use cases you listed. Something like:
let mut deserializer = serde_json::Deserializer::from_str(" { ... } ");
let defaults = MyStruct { ... };
let s: MyStruct = deserialize_with_defaults(&mut deserializer, &defaults)?;

I don't think that would be possible because serde::Deserialize only has one method and it returns Result<Self, _> this means that there is no way to inject a "base" object and no way to deserialize in a way that doesn't return a complete object.
The only way I can possible thing of this working is something like serialize the default object, prepend it to the data, then deserialize. However that is awful for a variety of reasons not which of least is that it won't work for most formats.
I think this would need to be supported in-crate. My first idea without much thought would be something like:
pub trait DeserializePartial<'de>: Sized {
    type Base;

    fn deserialize_partial<D: serde::Deserializer>(base: Self::Base, deserializer: D) -> Result<Self, D::Error>;
}

Ideally serde_derive could then be updated to generate the Base type automatically. It could be a struct that is a clone of the struct that it is developed on except that every field is wrapped in Option. (More efficient systems could also be used).

Actually, there is a hidden method in the Deserialize trait, but I don't think that this will help you much:
https://github.com/serde-rs/serde/blob/c3ce2c934a9123b5a309256d65cca6b6d19466ec/serde/src/de/mod.rs#L543-L570

Yeah, I think the "base" would need to be a different type in most cases. (An exception being where the struct can implement Default or has a similar default state.) I think the semantics are also expected to be different. I would expect this method to still return an error or set a field to the default if missing, not reuse the value of "place". It seems that the input value of place should not affect the semantic value of the output value of place.


[SPARK-50672][PYTHON][TESTS] Make openpyxl optional in PySpark Tests

What changes were proposed in this pull request?
Make openpyxl optional in PySpark Tests
Why are the changes needed?
openpyxl is an optional dependency of pandas, also optional to pyspark
the test should not fail without it.
Does this PR introduce any user-facing change?
no, test only
How was this patch tested?
manually check
Was this patch authored or co-authored using generative AI tooling?
no

merged to master


SSF load new table, not effected in  XLSX.read

I set XLSX.SSF.load('yyyy"年"m"月"', 52), add one customer code in SSF. I can use XLSX.SSF.get_table() to get new ssf table successfuly.  Then when I use this.wb = XLSX.read(data, {type: "binary"}), this.wb.SSF is old.
Is my use incorrect?
Waiting your response..
please feedback about this problem : (
Thanks!

@wang4869
I'm facing the same problem. Did you find out the reason and how to fix it?


TupleReturn does not work with static members.

/** @tupleReturn */
public static screenToMap(x: number, y: number): [number, number]

Compile to table return and unpack function.
_Originally posted by @dasannikov in https://github.com/render_node/MDEyOklzc3VlQ29tbWVudDQ0NDA2MjAwMQ==/timeline/issue_comment#issuecomment-444062001

Released with v0.12.0 f5ed6570d198a002c4fb91db54e59cdde3dbaf40


[WIP] chore: add additional args to client state GetLatestHeight interface method

Description
This PR is a work-in-progress. Looks like GetLatestHeight is used extensively throughout the codebase in test code.
closes: #XXXX
Commit Message / Changelog Entry
type: commit message

see the guidelines for commit messages. (view raw markdown for examples)

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

[ ] Targeted PR against correct branch (see CONTRIBUTING.md).
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[ ] Code follows the module structure standards and Go style guide.
[ ] Wrote unit and integration tests.
[ ] Updated relevant documentation (docs/) or specification (x/<module>/spec/).
[ ] Added relevant godoc comments.
[ ] Provide a commit message to be used for the changelog entry in the PR description for review.
[ ] Re-reviewed Files changed in the Github PR explorer.
[ ] Review Codecov Report in the comment section below once CI passes.

closing in favour of alternative approach using BeginBlocker


VMware ESX production checklist

Title
VMware ESX production checklist
Description
A number of large customers are running CockroachDB on VMware ESX.  Need to collect and document best practice:
Outline

NTP settings were discussed at https://github.com/cockroachdb/docs/issues/3716
CPU, RAM, IO allocation should be documented

Expected Audience
DBA, DevOps, Arch

@RaduBerinde
I would greatly appreciate your assistance with this project.


No way to get continuation token for TableClienmt.listEntities()

Package Name: @azure/data-tables
Package Version:  12.1.2
Operating system: Windows 10

To list the entries in a table one page at a time, you can do something like this:
const client = TableClient.fromConnectionString(
  STORAGE_CONN_STR,
  TABLE_NAME
);

const firstPage = await client
    .listEntities()
    .byPage()
    .next();

This works fine if you want to get all the pages at once, but one of the main reasons to getting entries by pages is if you want to come back and get them a few at a time, like when the user literally navigates to the next page. But, as far as I can tell, the JS API offers no easy way to get a page continuation token.
const nextPage = await client
    .listEntities()
    .byPage({
      continuationToken: , // <-- Where do I get this value from?
      maxPageSize: 10
    })
    .next();

This is weird because all the other Azure APIs allow you to do this. For example, in python, you could do something like:
generator1 = table_client.query_entities(results_per_page=10).by_page()
page1 = next(generator1)
entities1 = list(page1)

generator2 = table_client.query_entities(results_per_page=10).by_page(generator1.continuation_token)
page2 = next(generator2)
entities2 = list(page2)

The pages (i.e. the results of listEntities().byPage().next()) do have a nextPartitionKey and nextRowKey field (very weird in itself because they are in what is supposed to be an array and not in the TypeScript signature). I looked at the source code and copy+pasted the encodeContinuationToken() function and tried to call it with nextPartitionKey and nextRowKey to get a continuation token like this:
const client = TableClient.fromConnectionString(
  STORAGE_CONN_STR,
  TABLE_NAME
);

const page1 = await client.listEntities().byPage({ maxPageSize }).next();
const page2 = await client
  .listEntities()
  .byPage({
    maxPageSize: 10,
    continuationToken: encodeContinuationToken(
      page1.value["nextPartitionKey"],
      page1.value["nextRowKey"]
    ),
  })
  .next();

console.log(page1.value, page2.value);

The result is two totally identical pages with the same next partition and row values. encodeContinuationToken is giving a valid-looking b64 string, but it seems to be getting ignored.

@QuinnFreedman thanks for reaching out. We recently merged a change to expose the continuatioToken and enable byPage to set it. #18179
This change has not been release yet, we are planning to release next week. In the meantime you can try out the fix intalling our nightly build with npm install @azure/data-tables@dev.
Here is an example of how to work with continuation tokens in this SDK:
https://github.com/Azure/azure-sdk-for-js/blob/848242cd9d84470ec7c4b53a3ad30e16f1f8668b/sdk/tables/data-tables/samples-dev/usingContinuationToken.ts#L36-L44
https://github.com/Azure/azure-sdk-for-js/blob/848242cd9d84470ec7c4b53a3ad30e16f1f8668b/sdk/tables/data-tables/samples-dev/usingContinuationToken.ts#L50-L56
I would appreciate if you can give it a try installing the latest dev build, and let me know how it goes.

This works for me:
const continuationToken = req.body.continuationToken;

const page = await client
  .listEntities({
    queryOptions: {
      filter: "location eq 'location A'",
    },
  })
  .byPage({
    maxPageSize: 2,
    continuationToken: continuationToken ? continuationToken : null,
  })
  .next();


install Cairo before Bioc packages

proteomics containers build fine locally

@lshep - if you get a chance, could you trigger a proteomics build at some point. Thanks.

@lshep - please ignore my previous comment, I just realised now that I can trigger a build myself.


New update black screen.

Updated to 3.7.7 on foundry V9 build 255 and when the world loads in there is a black screen, the UI is still there but the map is black. Disabling PV fixes issue, reenabling causes same

.

Please let me know whether there are any error messages in the console (F12 -> Console).

New to this console stuff, is this what you are looking for?

I'm seeing the same error since that last update.

This is probably a cache issue. This error shouldn't be possible with 3.7.7. Try CTRL+F5. If that doesn't work, uninstall and then reinstall PV.

Removing and reinstalling fixed the issue, thanks for the amazingly rapid service!!!!

Also solved my problem. Thanks!


[stable/traefik] Fix invalid template variables

Currently the chart fails while rendering templates if ACME option is enabled.  This fixes the issue/typos.

I signed it!

Nice catch. Just bump the version in Chart.yaml to 1.1.2-g and then this LGTM.
cc @viglesiasce @lachie83 we should merge this one as soon as possible since it's a big fix.

Version bumped, thanks for quick review.

Thanks @bzub!

@k8s-bot ok to test

Thanks @bzub. @krancour I'll work on getting this merged.


QuickStack exception on latest update (v0.216.9);

Quick summary: Using QuickStack without a container open doesn't appear to do anything.
Game Version: v0.216.9
Game state: Local w/out dedicated server
Mods:
BepInEx: v5.4.2105
BepInEx Configuration Manager: v18.0.0
Quick Stack Store: v1.3.10
Debug steps:

Ran game in Vanilla state, saved. Re-loaded with select mods above.
All settings default/fresh install
Changed default keybind from P to BackQuote
Changed QuickStackTrophiesIntoSameContainer to Enabled

Full Summary:
Clicking the Quickstack icon without a container open, or using the shortcut doesn't do anything. When checking the Console log, the following error is displayed using the shortcut:
[Error  : Unity Log] MissingMethodException: long ZNet.GetUID()
Stack trace:
QuickStackStore.QuickStackModule.QuickStackIntoMultipleContainers (System.Collections.Generic.List`1[T] trophies, System.Collections.Generic.List`1[T] nonTrophies, Player player, System.Collections.Generic.List`1[T] containers) (at <7038b2bb98ca4ab0b7829a4bfa31c245>:0)
QuickStackStore.QuickStackModule.DoQuickStack (Player player, System.Boolean QuickStackOnlyToCurrentContainerOverride) (at <7038b2bb98ca4ab0b7829a4bfa31c245>:0)
QuickStackStore.KeybindChecker+Player_Update_Patch.HandleGenericKeys (Player player) (at <7038b2bb98ca4ab0b7829a4bfa31c245>:0)
QuickStackStore.KeybindChecker+Player_Update_Patch.Postfix_Patch (Player __instance) (at <7038b2bb98ca4ab0b7829a4bfa31c245>:0)
(wrapper dynamic-method) Player.DMD<Player::Update>(Player)

The following error displayed when using the Q button from the inventory window:
[Error  : Unity Log] MissingMethodException: long ZNet.GetUID()
Stack trace:
QuickStackStore.QuickStackModule.QuickStackIntoMultipleContainers (System.Collections.Generic.List`1[T] trophies, System.Collections.Generic.List`1[T] nonTrophies, Player player, System.Collections.Generic.List`1[T] containers) (at <7038b2bb98ca4ab0b7829a4bfa31c245>:0)
QuickStackStore.QuickStackModule.DoQuickStack (Player player, System.Boolean QuickStackOnlyToCurrentContainerOverride) (at <7038b2bb98ca4ab0b7829a4bfa31c245>:0)
QuickStackStore.ButtonRenderer+MainButtonUpdate+<>c.<UpdateInventoryGuiButtons>b__0_6 () (at <7038b2bb98ca4ab0b7829a4bfa31c245>:0)
UnityEngine.Events.InvokableCall.Invoke () (at <80fe1a4e36fe44618284312cb721a597>:0)
UnityEngine.Events.UnityEvent.Invoke () (at <80fe1a4e36fe44618284312cb721a597>:0)
UnityEngine.UI.Button.Press () (at <e97aac8d17514bea83fadfd039e9187a>:0)
UnityEngine.UI.Button.OnPointerClick (UnityEngine.EventSystems.PointerEventData eventData) (at <e97aac8d17514bea83fadfd039e9187a>:0)
UnityEngine.EventSystems.ExecuteEvents.Execute (UnityEngine.EventSystems.IPointerClickHandler handler, UnityEngine.EventSystems.BaseEventData eventData) (at <e97aac8d17514bea83fadfd039e9187a>:0)
UnityEngine.EventSystems.ExecuteEvents.Execute[T] (UnityEngine.GameObject target, UnityEngine.EventSystems.BaseEventData eventData, UnityEngine.EventSystems.ExecuteEvents+EventFunction`1[T1] functor) (at <e97aac8d17514bea83fadfd039e9187a>:0)
UnityEngine.EventSystems.EventSystem:Update()

I ran into this too on the latest build.

It's worth noting even the original Quick Stack mod is getting a similar bug.

I have this fixed for everything but the trash button in the UI. If I can get that fixed I'll do a pull request to incorporate the changes.
It's all just because they changed the method names for a ton of things (and how to get a UID)

did the update fix all these issues?


Dictionary leaves out common words

The dictionary oddly leaves out common words like "cow" (tho "cows" seems to be in the dictionary). I recommend including a much broader dictionary step. I would imagine that actual password cracking techniques will go through words in order of assumed commonality. This vastly overestimates the security of passwords that contain words this dictionary set doesn't happen to contain. It would be best to contain a more realistic set of dictionary words.

Yeah, the "English Wikipedia" word list is weird. What dictionary do you think should be used instead?

Well, a list like one of the following might be ideal:

https://github.com/first20hours/google-10000-english
https://www.ef.edu/english-resources/english-vocabulary/top-3000-words/

Well, a list like one of the following might be ideal:

https://github.com/first20hours/google-10000-english
https://www.ef.edu/english-resources/english-vocabulary/top-3000-words/


Latest IEM Final nuke demo anaylzing failing

demo link: https://www.hltv.org/download/demo/73074
If you try to analyze "nuke" demo it fails to go beyond round 2. which cause view player to not work correctly. first it analyzes  all round and then on second loop it stops after round 2.
NO error messages or logs.

A new major version has been released, closing the issue 🧹


update the ci docs to use the pro image

Can we also mention that we now offer localstack/localstack-pro:2.0.2-bigdata?
They should know that they can use this image in the CI environment to make their lives easier.

Can we also mention that we now offer localstack/localstack-pro:2.0.2-bigdata? They should know that they can use this image in the CI environment to make their lives easier.

I think that's a special use-case that we can document elsewhere.


Allow RE to execute sub plans in parallel

Currently the run engine does one task at a time. There are use cases where we would like to ask the run engine to "run this plan/these plans and tell me when you are done". Example use cases are:

Running alignment or calibration procedures which are independent of each other while also preparing a robot to select a new sample
Change beamline energy while a sample robot get's a new sample
Move/Spin a sample in a given trajectory while running a measurement technique
Arm multiple detectors at the same time
https://bliss.gitlab-pages.esrf.fr/bliss/1.10.x/motion_hook.html

In each of these cases there are different ways to solve the problem. Execution could be handled in Ophyd, or in EPICS/Tango/etc layer or in the device itself. This is discussed in https://github.com/bluesky/ophyd/issues/1121#issuecomment-1579146876. @tacaswell touched on the problem some time ago in https://github.com/bluesky/bluesky/pull/541
Suggested reading by @tacaswell: https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/
Proposed Changes

A new message added to the RE which takes as arguments a plan and a set of verbs that are allowed to be used. This allows us to have some control over what plans are allowed and not. async def _run_sub(self, plan, verbs) should return a status object
RE should be modified to be able to run sub_plans in parrallel in a way that is backwards compatible.

As discussed, for a lot of these cases we can get away with a very small subset of verbs that the _run_subs is allowed. I suggest if whoever does the implementation finds that it's more complex to allow the verbs that interact with the Event system they spin those into a new issue

The Msg should return a status object to the calling plan and it should particpate in the same wait logic that set and trigger do so you can write things like:
def my_plan():
   ...
   yield from run_sub_plan(adjust_energy(), group='setup')
   yield from run_sub_plan(change_sample_with_robot(), group='setup')
   yield from bps.set(detector_pos, 5, group='setup')
   yield from bps.wait(group='setup')
   ...
   yield from bp.count(...)

other design questions:

should it take a called plan (no need to get involved in args/kwargs, but might lead to sticky situations with rewind), insist on taking a no parameter callable (still no arg/kwargs forwarding, and gives the option of not doing "rewind" in the sub-plans by saying it will just run the sub-plan again if replayed), or do the full args/kwargs forwarding.  asyncio went with the "parameter less" function option (and says "use functools.partial if you want to do the args/kwargs binding") so that is probably the right choice here.
cleaning up any running tasks on return is straight forward (we already have logic to call stop on everything we touched so it should just be extending that), but we may want to add a way to cancel these explicitly or rely on try...except to get the scoping right (so only kill them if something else goes wrong / we are interputed).  I think we do want a clean way to cancel them so that you can write infinite subplans and then gracefully clean them up.
how to handle failures in the sub-plans, both from a "how do we get a good traceback to the user" and "how do we abort everything else point of view".  For the second I think we can follow the model for how we handle failures on set
lament that we do not have gwith to complete the set of with and awith (not really a design choice.  The problem is you can not write a context manager that will yield on cleanup.  I had some conversations a long time ago with core developers and they did not think we could sell Guido on adding gwith from a completionist point of view and did not think our use case was broad enough to warrant a new keyword in the language. Maybe the landscape on this is different now).  The reason it is worth lamenting this here is that may prevent us from using context managers as nicely as you can in the trio context.

For a while I've been playing with the idea of a special plan that accepts other plans and runs all of them simultaneously by "intermixing" the messages fed to the RE. Finally sat down to try writing such a plan. I think I like this pattern, sharing here in the unlikely case that others find it useful.
import uuid
import bluesky
import collections
from dataclasses import dataclass, field
from bluesky import Msg
from bluesky import plan_stubs as bps
import ophyd
from ophyd import sim, status

@dataclass
class StatefulPlan:
    plan: object
    statuses: list = field(default_factory=list)
    waiting: bool = False

class WaitLocalizer(object):

    def __init__(self, *args):
        self.sps = list()
        for plan in args:
            sp = StatefulPlan(plan=plan)
            self.sps.append(sp)
        self.uuid = uuid.uuid1()

    def __iter__(self):
        while True:
            for sp_index, sp in enumerate(self.sps):
                if sp.waiting:
                    continue
                while True:
                    try:
                        msg = next(sp.plan)
                    except StopIteration:
                        self.sps.pop(sp_index)
                        break
                    if msg.command == "wait":
                        sp.waiting = True
                        break
                    print(msg.command, msg.obj.name, msg.args)
                    ret = yield(msg)
                    if isinstance(ret, bluesky.protocols.Status):
                        sp.statuses.append(ret)
                        ret.add_callback(self.status_callback)
            if len(self.sps) == 0:
                print("returning")
                return
            yield Msg("trigger", self, group=self.uuid)
            print("waiting")
            yield Msg("wait", group=self.uuid)

    def status_callback(self, *args, **kwargs):
        for sp in self.sps:
            for index, status in enumerate(sp.statuses):
                if status.done:
                    sp.statuses.pop(index)
            if len(sp.statuses) == 0:
                sp.waiting = False
        try:
            self.status.set_finished()
        except ophyd.utils.errors.InvalidState:
            pass

    def trigger(self):
        self.status = status.Status()
        return self.status

# slow down motors
sim.motor1.delay = 2
sim.motor2.delay = 0.321

def plan1():
    for x in range(-10, 10, 2):
        yield from bps.mv(sim.motor1, x)

def plan2():
    for x in range(0, 100, 5):
        yield from bps.mv(sim.motor2, x)

def plan():
    yield Msg("open_run")
    yield from WaitLocalizer(plan1(), plan2())
    yield Msg("close_run")

RE = bluesky.RunEngine()
RE(plan())

The basic idea is

pass through messages from the subplans to the RE and collect status objects as you see them---keep track of witch statuses came from which subplans
if you see a wait message, intercept it---don't pass it to the RE
keep passing messages until all the plans are in a waiting state, then have the RE wait on a special status that you control
once any of the subplan status objects you've collected finish, release special status and RE will continue consuming messages from any plans that aren't waiting

It does work, for a limited set of plans... Mostly a personal learning exercise... To work properly, WaitLocalizer would need to grow proper support for sleep and respect for groups of status objects. I'm also pretty sure there's a race condition if the subplan statuses finish too quickly, which needs to be accounted for.

@whs92 Yes, that is the gist of it.  Once we start the N sub-plans we need to make sure that they either finish or are cancelled no matter what else happens and getting that right is subtle.
For context, a simpler version of this problem came up with the caproto servers which have a bunch of background tasks that need to be canceled on exit.  With trio I think we got it right on the first try, with asyncio it took us four or five tries to get it right.

Yeah that article has a lot in it, it took me a couple passes to follow it. Specifically I think the lesson for us is, asyncio makes it easy to do cancellation wrong.

One potential use case that might help; we're running a plan at the moment that triggers does a count([detector, beamstop], 100) basically, at the same time we want to monitor a spectrometer, so we use monitor_during_wrapper(count[detector, beamstop], 100), [spectro]).
However, we need to trigger and read from the spectro in a different cadence to the detector, so atm I asyncio.create_task a background task to trigger the spectro every 0.x seconds, while the main task is monitoring the spectro.
Ideally, I could just have a separate plan that triggers and reads the spectro, that runs side by the side the count([detector, beamstop]) plan.

I'm going to have a go at implementing some very basic version of this.
I think a basic list of allowed verbs could be:
ALLOWED_VERBS = {
    "null", "stop", "set", "trigger", "sleep", "wait", "prepare",
    "collect", "kickoff", "complete", "stage", "unstage",
}

Can anyone identify any obvious serious issues with running these in the background? The idea is that the "main thread" would still be able to interact with the event system, and this still ought to cover all the use cases covered in the issue description.
I'm also interested in what people think should happen if a close_run happens (for an outermost run) while any subplans are running? Should we:

a. error, tell the user they must explicitly wait?
b. block and wait?
c. cancel them and warn?
d. something else?

My thought had been to do with with tasks (so each sub-plan gets its own task and can context switch at any await).  This might allow too much switching, but only context switching on wait (and sleep?) seems like too little.
The simplest thing on rewinding is to ban it while sub-plans are running (if you cache the "launch a subplan" message and the messages the subplan generates on rewind you will duplicate them.   We have a preview of some of this complexity comes from "interlace all the plans in the main _run" from suspenders (which have the ability to add a plan to the stack of plans) and I'm worried about it!
Can a sub-plan launch sub-plans?  If so, if you cancel a sub-plan how do you make sure it and its children (but not any others) get canceled?
Given the amount of state _run holds I'm worried about the complexity if making each of those an array when you get the same effect (via a new locals()) from the task-based approach and let Python do the book-keeping for us.
If something like what @untzag proposed is sufficient, then I think that should go into bluesky.preproessors more-or-less as is without having to touch _run.

As for what to do about close_run, consider something like:

def jogger(motor, start, stop):
    for target in itertools.cycle([start, stop]):
        yield from bps.mv(motor, target)
        yield from bps.checkpoint()

def main_plan():
    token = yield from run_subplan(jogger(motor, start, stop))
    for j in range(15):
         yield from bp.count(...)
    yield from abort_subplat(token)

should be possible as should
def background(det, delay=5):
    while True:
        yield from bps.trigger_and_read([det], stream='background')
        yield from bps.checkpoint()
        yield from bps.sleep(delay)

def main_plan():
    yield from bps.open_run()
    token = yield from run_plan(background(det1))
    for j in range(15):
        yield from bps.mv(mtr, j)
        yield from trigger_and_read([other, dets])
        yield from bps.checkpoint()
    yield from abort_run(token)
    yield from close_run()

I suspect that we need to be careful about managing the bundlers carefully so that if the run is closed, the bundlers go away and the second case would fail "naturally".
I think on _run exit we have to make everything stop, but that is different than close_run.

My thought had been to do with with tasks (so each sub-plan gets its own task and can context switch at any await).

I don't really understand what would be executing on the messages in this case, do you mean creating a new RE instance and creating a task out of partial(RE, subplan_generator, is_subplan=True)?

The simplest thing on rewinding is to ban it while sub-plans are running

Yes, that was what I meant

Can a sub-plan launch sub-plans? If so, if you cancel a sub-plan how do you make sure it and its children (but not any others) get canceled?

No, I wasn't going to allow run_subplans in subplans - no use case we have imagined so far would need it, and as you say it introduces a lot of extra complexity

Given the amount of state _run holds I'm worried about the complexity if making each of those an array when you get the same effect (via a new locals()) from the task-based approach and let Python do the book-keeping for us.

sure, I agree; happy to try out both approaches and compare them - but I wasn't going to make all of the RE state into arrays, since the subplans don't need a vast majority of it, just add a new class to manage the minimal state that they do need. the only change I'd hoped to make to _run was where to pull the next message from

After discussing this with @DominicOram @callumforrester etc. we're convinced that https://github.com/bluesky/ophyd/issues/1121#issuecomment-1579146876 / https://github.com/bluesky/bluesky/issues/1654 is indeed the better way to approach this, so I will have a look at doing that first

no use case we have imagined so far would need it

Say you have a subplan to do some non-trivial physical reconfiguration of the beamline (move a sample through a load lock, move a sample from a sample library to being mounted for measurement, moving a bunch of mirrors + optics to change which end station is going to get beam, moving optics + detectors to change what technique is being used etc) and you want to be able to factor the steps of that nicely.
def subplan_to_change_optics():
    ...

def subplan_to_change_detectors():
    ...

def subplan_change_technique():
    yield from run_subplan(subplan_to_change_optics)
    yield from run_subplan(subplan_to_change_detectors)
    yield from bps.wait()

def load_sample(sample):
    ...

def do_measurement(sample):
    yield from run_subplan(load_sample, sample)  # however we spell passing parameters through
    yield from run_subplan(subplan_change_technique)
    yield from bps.wait()
    yield from take_data()

There may be things we want to prevent subplans from doing (initially I thought "taking data" would be one, but the use-case from the Australian's needs that) but I suspect that those will be things that are likely to mess with other running (sub) plans (like unregistering a message or callback).   Anything we don't support on the first pass we should at least think through if it could conceptually be supported and make sure we do not make it impossible in the future (even if we do not do that work now).


Question about worker thread in GPipe

Hi, thanks for the fantastic work!
I have a question about the micro-batch lockstep.
https://github.com/kakaobrain/torchgpipe/blob/14bbc9befee5f6169ca559e61026e1e99efcfbd6/torchgpipe/gpipe.py#L367
In the comment it says "During this partition is executing a micro-batch, to copy a micro-batch by the next partition would be blocked". Why would dumping the message into the queue be blocked by the next partition thread?

Hi, thanks for diving into our code.
When we copy a PyTorch tensor from a GPU to another GPU, the copy might be delayed until both GPUs finish to execute all scheduled CUDA kernels. In other words, a GPU-to-GPU tensor copy requires to synchronize both GPUs.
GPipe enforces users to design well-balanced partitions to achieve optimal performance. So we can assume every partition has similar computational cost. But one partition doesn't produce only one CUDA kernel. Each partition produces several CUDA kernels. Even the total kernel cost per partition is almost identical, the cost of each kernel may be jagged.
The lockstep approach ensures to register copy commands behind the final kernel of each partition, rather than the middle of a partition. This deterministic behavior reduces the frequency of delayed copies. The below diagram compares how the lockstep minimizes the delayed copies:

Furthermore, the time to compute the first micro-batch by latter partitions is more important than the time to compute the last micro-batch by prior partitions. The lockstep approach makes the GPUs understand this priority explicitly.
I also attached the timeline comparison:

Thanks a lot for the detailed explanation!
For the timeline comparison, what are these blue and red represented though?

Blue and red means computation and copy, respectively. It was captured from NVIDIA Nsight Systems, which is a visual CUDA profiler.

I see, thank you so much!


added more info about technologies used to write smart contracts

Added section for technologies used to write smart contracts in Ethereum and Hyperledger.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Looks good @dipakyadav! Congratulations on your first contributing to freeCodeCamp!


Change the ClassMetadataFactory

Change the ClassMetadataFactory to LazyLoadingMetadataFactory because it is deprecated in last version.

Thank you @juliosbonifacio.


Lua language support

I know this might be a long shot, but polyglot notebooks are really useful, and I use lua for game programming, so it would be a match made in heaven to be able to use lua as a language in the notebooks. Any chance of it ever happening?

This is a very nice question. This is something that could be started as extension and plugin. having lua support is one thing, how would you envision the interaction with the game development?

+1 on this issue.
We can fork moonsharp and upgrade it to 5.4LUA
Or just use native bindings

Yes, Lua seems to be the obvious one missing from the list of supported languages.


contenttypes/ListParts  list classify ？？？？

Can I classify the Li list of Content Definition/contenttypes&ListParts? So many lists are very messy!

There is no way to organize content types right now.

Thank you for your enthusiasm and answers。

No, there is no way to do it right now.

Virtual UI? No actual path! Just the user interface display? Just like a drop down menu(☆^ー^☆)


.Net Fix Add Missing OpenAI Connector Choice properties to Metadata

Description
Resolves #5289
This pull request includes changes to both the dotnet/samples/KernelSyntaxExamples/Example43_GetModelResult.cs and dotnet/src/Connectors/Connectors.OpenAI/AzureSdk/ClientCore.cs files. The changes mainly focus on improving the handling of metadata and simplifying the codebase.
Changes in Example43_GetModelResult.cs:

The method GetTokenUsageMetadataAsync() was modified to use implicit type (var) instead of explicit type (Kernel) when creating a kernel. This change simplifies the code and makes it more readable.
A new method GetFullModelMetadataAsync() was added. This method creates a kernel, defines a function, invokes the function through the kernel, and displays the results. This addition expands the functionality of the class.

Changes in ClientCore.cs:

The method ClientCore() was modified to use GetTextChoiceMetadata() instead of GetChoiceMetadata(). This change improves the handling of metadata.
The method GetStreamingTextContentsAsync() was also modified to use GetTextChoiceMetadata() instead of GetChoiceMetadata(). This change improves the handling of metadata.
The methods GetTextChoiceMetadata(), GetChatChoiceMetadata(), and GetResponseMetadata() were modified to include additional metadata fields. This change improves the amount of information available in the metadata.
The method AddResponseMessage() was modified to handle null values for finishReason. This change improves the robustness of the code.

Changes in AzureOpenAIChatCompletionServiceTests.cs:

The method GetStreamingChatMessageContentsWorksCorrectlyAsync() was modified to use an enumerator instead of a foreach loop. This change simplifies the code and makes it more readable.

Changes in chat_completion_streaming_test_response.txt:

The test response was modified to include additional data. This change improves the accuracy of the tests.

@RogerBarreto do you think about introduce same pattern for matadata like is in Gemini and HuggingFace? It would be nice have inteliisense support.

@Krzysztof318 That's a good suggestion, not in the plans on changing the current OpenAI connector as some other updates will come from Azure on that regard.
We will eventually splitting Azure from OpenAI specific connectors.


Adding video tutorial

Hi!
Have you uploaded the video tutorial shown in https://www.egi.eu/event/egi2022/ somewhere?
It would be nice to have it linked in https://github.com/grycap/oscar-cli/blob/main/README.md
Thanks!

Hi Sebastián,
Sure thing. The video is now available in: https://youtu.be/itaKJyd9QX4
We will include it in the documentation of OSCAR, probably in https://docs.oscar.grycap.net
Cheers.

Thanks!
Could you also link https://docs.oscar.grycap.net/ from https://github.com/grycap/oscar-cli/blob/main/README.md?
Best regards,
Sebastian


CU1280 GetAndAssignValue function

DearTeam Member,
Could you change the GetAndAssignValue procedure from local to global?

This procedure already made global in v.22
procedure GetAndAssignValue(var CurrencyExchangeRecordRef: RecordRef; FieldNo: Integer; DefinitionDataExchField: Record "Data Exch. Field"; DefaultValue: Variant): Boolean
var
    DataExchField: Record "Data Exch. Field";
    DataExchFieldMapping: Record "Data Exch. Field Mapping";
    UseDefaultValue: Boolean;
begin
    UseDefaultValue := not GetFieldValue(CurrencyExchangeRecordRef, DataExchField, DefinitionDataExchField, DataExchFieldMapping, FieldNo);
    exit(AssignValue(CurrencyExchangeRecordRef, DataExchField, DataExchFieldMapping, UseDefaultValue, DefaultValue, FieldNo));
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: .


Add ClusterPolicy controller and automatically exclude chart-operator from custom policies

This PR adds the functionality needed to exclude chart-operator ServiceAccount from custom policies automatically.
The new clusterpolicy controller will check for ClusterPolicy changes and create a custom PolicyException based on the list of Kinds defined in the chartOperatorExcemptedKinds value.
If no Kinds are defined, the controller does not run.

@fhielpos I like the hybrid but Excempted should be either Exempt (my vote) or Excepted


Paste with syntax highlighting is not pasting empty lines

VSCode Version: VS Code Insiders from 2/17/17
OS Version: Windows 10

Steps to Reproduce:

Write some code in a syntax highlighted language like PowerShell which has empty lines between real lines:

Set-StrictMode -Version Latest
$ErrorActionPreference='stop'

Import-Module "$PSScriptRoot\DSCEngineering\helpers\LogUtility.psm1"
Import-Module "$PSScriptRoot\DSCEngineering\helpers\ToolUtility.psm1"

# In case of DevWorkflow, this script will be bin placed to <DscRepoRoot>\build\x64fre\bin\Tools
# Workflow for generating package to be handed over to OMS

Copy the snippet of code and paste it into a new email in Outlook.  The result looks like this:

Notice that the two blank lines are not retained.
/cc @rebornix

This is when HTML eats you plain text line break or whitespaces :(

Awesome, thanks!


Do not work with Admin 3.6.2

Adapter is not working anymore up from Admin3.6.2. Up to 3.6.0 everything was fine.
Thanks for your help!

What exactly is not working?

Nothing :-D. The adapter do not start. I fixed it now at the same way the soef adapter tr-064 could have been fixed before you fixed it now as the community adapter of it:
cd /opt/iobroker/
iobroker stop js2fs
cd /opt/iobroker/node_modules/iobroker.js2fs
npm install soef

So had nothing to do with admin ?

I think the problem belongs to new Admin 3.6.2 because at 3.6.0 everything was fine.
I had the same problem on tr064 so I tried the same to fix it and it works:
https://forum.iobroker.net/topic/18412/tr-064-0-adapter-fehlermeldung/7
But if I untetstand it correctly, this method is only a workaround so after it I installed your new tr064 community adapter. So I think the js2fs adapter has to be updated also to run without this workaround

The Community Adapter is currently not different to GitHub version of original one. We did not had time so far to adopt it.
Reason is also not Admin, it is "npm". Sometimes npm reorganizes the packages and it seems that this can cause this problem somehow.

Ah, ok. So for the meantime I have a solution and waiting for your update :-)


BUNDLED WITH updates a lower version as well.

With reference to the locked #3697
@indirect Wrote this:
Older versions of Bundler will not lower the number in the lock, but will warn the user that they are out of date and may not be able to install this bundle successfully.

However I can clearly see a downgrade of Gemfile lock from 1.10.4 to 1.10.3 in one of our developers machines, I have not spent time debugging that as I first hoping for clarification.
Isn't that contradicting the above intended behaviour?
As an aside, thanks for making Bundler !! :smile_cat:

Can you provide a reproduction script for that, since it shouldn't ever happen?

No. I can't. It happened once and we couldn't recreate it. I'll have to assume some rvm/bundler hiccup, but that's just a wild guess.
Closing this, 10x!

A possible answer is that that person had a ruby with Bundler 1.9 installed, ran a command, realized they were on the wrong Ruby, than switched to a Ruby with 1.10.3 installed.
To be super clear: that should not happen. and if it does it is a bug we will give high priority. 😅


Entities.EscapeMode.none question

Hi @jhy and first of all thanks for the amazing work with the library.
I'm facing a situation where I would need the Entities.EscapeMode.none option.
In this pull request you say:

If you want plain text output, use one of the .text() methods. If one of those doesn't fit the use case, I'd be happy to hear more about the use case and we can explore ways to improve those.

I was using .text() until now, but I found a weird behaviour. If the input is something like:
<script src=\\\"https://xss.cacker.io/\\\">removed script </script>
Then JSoup will include it in the document, and when extracting the text we will get:
<script src=\"https://xss.cacker.io/\">removed script </script>
This 1) turns encoded-non dangerous input to input that could be dangerous, and most importantly 2) modifies user input. My goal would be to sanitize input but modify it as little as possible and only based on the data in the SafeLists. Data is not always outputted after using Jsoup, but sometimes also saved into a DB, and having encoded text is not desirable.
I know JSoup tries to do both output encoding (with valid HTML) and input sanitization, but in my case I only need Input Sanitization, so it would be really great to have a flag that allows me to do that.
What could be a way to solve this issue I have?
Thanks in advance.

But if you are taking the text() and storing it in a DB, and then show it to a user in a HTML context, you should escape the entities on output (because it's text, not HTML).
Or, use the html() and then you won't need to escape them. So, I think that is the path that most closely meets your ask - it's only doing sanitization.
If you treat text as text, and HTML as HTML, there is no need for a .none option. I won't ever add it.

The cleaner does clean. If you give it HTML that includes a script tag, it will remove it. <script> is not a script tag and has no danger to users. It is a textnode in HTML.


Pipe argument not working

Hi,
I have created custom pipes for filtering data. Normal pipe and custom pipe working fine. But the issue is pipe argument not working.
My custom pipe code is:
import {Pipe} from 'angular2/core';
@Pipe({
  name: 'toggl'
})
export class ToggleArgs {
  transform(value,[status]) {
    return value.filter((item)=>item.status === status);
  }
}

My component code is
import {Component} from 'angular2/core';
import {ToggleArgs} from '../../pipes/toggle-args/toggle-args';

@Component({
  selector: 'test-component',
  templateUrl: 'app/components/test-component/test-component.html',
  styleUrls: ['app/components/test-component/test-component.css'],
  providers: [],
  directives: [],
  pipes: [ToggleArgs]
})
export class TestComponent {
list;
 constructor() {
    this.list = [{"title": "eat", "status": "WIP"},{"title": "code", "status": "WIP"},{"title": "sleep", "status": "WIP"}]
  }
}

html code:
<li *ngFor="#item of list | toggl:'completed'">{{item.title}} and {{item.status}}</li>
Output:
Build error
[DiffingTSCompiler]: Typescript found the following errors:
app/pipes/toggle-args/toggle-args.spec.ts (22, 14): Supplied parameters do not match any signature of call target.
Same code I have tried  as per the quickstart guide (https://angular.io/docs/ts/latest/quickstart.html) it was working fine. Please solve this issue.

The error you're getting is in the unit test for your pipe (toggle-args.spec.ts). You probably changed the pipe but not the test, thus breaking the test.
You can either update the test, or remove it. In both scenarios your issue will be fixed.

Sorry i can't get you. Can you please share code. I have tried but still its not working. but this same code is working fine in angular quick start app

I would believe that you created your pipe via ng generate pipe toggl and by doing so there would have been 2 files created toggl.ts and toggle.spec.ts @filipesilva is suggesting, and I agree, that you have updated toggl.ts but not the test spec file to accommodate the code changes made.
If that is true then your spec file looks like this:
it('should transform the input', inject([Toggl], (pipe:Toggl) => {
  expect(pipe.transform(true)).toBe(null);
}));

You will have to update the expect statement to test what value you are supplying versus the result which is returned from your pipe

okay Brocco thanks

Hi brocco, i understand it. but js file is created automatically based on ts file right. then why i need to write manually.  Please make this process automatically. This will be helpful.

Well, when you do ng generate pipe toggl you get two files - a src/app/pipes/toggl.ts and src/app/pipes/toggl.spec.ts, right? The pipe itself and the unit test for it. They were generated for you at this point, but further changes to these files are on you.
Then you changed src/app/pipes/toggl.ts to your code - but didn't change src/app/pipes/toggl.spec.ts. The unit test code is still testing the old pipe code, and that fails.
So now you have two options: either you rework src/app/pipes/toggl.spec.ts to actually test your new pipe code, or you remove the test.

ok thanks. i will rework.

Please tell me if it works well for you. There isn't a lot of documentation on Angular 2 unit tests right now and sometimes it's hard to get them to work.


Fix minor typo

Checklist

[x] I've looked at the contribution guidelines.
[x] This pull request is complete and ready for review.

Cheers


[Feature] Deletion is not allowed when resources or udfs are used

If the resources or pdfs being used is deleted by mistake, the task execution will fail. I think it's unreasonable.
So we need to check that the resource has a task in use before we delete it.

If it's necessary, I will implement.


[Broken] Oreilly download support not working anymore and the playlist items give 404

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running yt-dlp version 2021.07.24
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

D:\Utils\YoutubeDL\DownloadOreilly>python "D:\Utils/YoutubeDL/ytdlsrc/yt-dlp/yt_dlp/__main__.py" --config-location "D:\Utils/YoutubeDL/OreillyConf/youtube-dl.conf" --playlist-start 1 --download-archive archive.txt --cookies cookies.txt
[debug] Custom config file: D:\Utils/YoutubeDL/OreillyConf/youtube-dl.conf
[debug] Custom config: ['-u', 'PRIVATE', '-p', 'PRIVATE', '-2', '-i', '-c', '-v', '--no-warnings', '--console-title', '--batch-file=batch-file.txt', '--write-annotations', '--write-description', '--write-info-json', '--write-thumbnail', '--sub-lang', 'en', '--write-auto-sub', '--write-sub', '--add-metadata', '--embed-thumbnail', '-o', '%(playlist_title)s/%(playlist_index)s. %(title)s.%(ext)s', '-f', 'bestvideo[height<=720]+bestaudio/best[height<=720]/worst', '--merge-output-format', 'mp4', '--mark-watched', '--geo-bypass']
[debug] Command-line config: ['--config-location', 'D:\\Utils/YoutubeDL/OreillyConf/youtube-dl.conf', '--playlist-start', '1', '--download-archive', 'archive.txt', '--cookies', 'cookies.txt']
[debug] Batch file urls: ['https://learning.oreilly.com/videos/learning-path-delivering/9781491989012/']
[debug] Loading archive file 'archive.txt'

[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.07.24 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: 11cc45718
[debug] Python version 3.9.1 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg n4.4-4-gacb339bb88, ffprobe n4.4-4-gacb339bb88
[debug] Proxy map: {}
[safari:course] Downloading login page
[debug] [safari:course] Extracting URL: https://learning.oreilly.com/videos/learning-path-delivering/9781491989012/
[safari:course]<PHONE_NUMBER>012: Downloading course JSON
[download] Downloading playlist: Learning Path: Delivering Applications with Docker
[info] Writing playlist metadata as JSON to: Learning Path - Delivering Applications with Docker\0. Learning Path - Delivering Applications with Docker.info.json
[safari:course] playlist Learning Path: Delivering Applications with Docker: Collected 74 videos; downloading 74 of them
[download] Downloading video 1 of 74
[safari:api] Downloading login page
[debug] [safari:api] Extracting URL: https://learning.oreilly.com/api/v1/book/9781491989012/chapter/video301824.html
[safari:api]<PHONE_NUMBER>012/video301824: Downloading part JSON
[safari] Downloading login page
[debug] [safari] Extracting URL: https://learning.oreilly.com/library/view/learning-path-delivering/9781491989012/video301824.html
[safari]<PHONE_NUMBER>012-video301824: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
  File "D:\Utils\YoutubeDL\ytdlsrc\yt-dlp\yt_dlp\extractor\common.py", line 679, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "D:\Utils\YoutubeDL\ytdlsrc\yt-dlp\yt_dlp\YoutubeDL.py", line 3151, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 561, in error
    return self._call_chain(*args)
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Users\mh-sh\AppData\Local\Programs\Python\Python39\lib\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

These same configs worked just 2 days ago, now I can't download any Oreilly content, I did try to check the links, they do give 404 from a browser.

I did try to check the links, they do give 404 from a browser.
That means the links are dead. Test with a known working link.

@Ashish0804 I think they meant the video links given by yt-dlp gave a 404. The playlist link seem to be fine.

@MinePlayersPE If you have an subscription, then can u test the command given by OP so we are sure it's a broken site?

The link works fine, I meant the items fetched by the link seem to give 404, the individual videos I mean.
I did test multiple courses I downloaded just 2 days ago, but it does not work anymore

Also to add, before I give credentials it could fetch 1min of each video, so even in the logged-out state it could download 1min, now it can't even find the video

It's seems to be working for me
yt-dlp -F https://learning.oreilly.com/videos/learning-path-delivering/9781491989012/
[safari:course]<PHONE_NUMBER>012: Downloading course JSON
[download] Downloading playlist: Learning Path: Delivering Applications with Docker
[safari:course] playlist Learning Path: Delivering Applications with Docker: Collected 74 videos; downloading 74 of them
[download] Downloading video 1 of 74
[safari:api]<PHONE_NUMBER>012/video301824: Downloading part JSON
[safari]<PHONE_NUMBER>012-video301824: Downloading webpage
[Kaltura]<PHONE_NUMBER>012-video301824: Downloading webpage
[Kaltura] 0_wuyu5ime: Downloading video info JSON
[Kaltura] 0_wuyu5ime: Checking mp4-409 URL
[Kaltura] 0_wuyu5ime: Downloading m3u8 information
WARNING: [Kaltura] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[info] Available formats for 0_wuyu5ime:
ID      EXT RESOLUTION FPS |  FILESIZE    TBR PROTO  | VCODEC    VBR ACODEC     ABR MORE INFO
------- --- ---------- --- - ---------- ----- ------ - ------- ----- --------- ---- ---------
mp4-56  mp4 audio only 0   | ~689.00KiB   56k http   |               unknown    56k isom
hls-58  mp4 audio only     |              58k m3u8_n |               mp4a.40.2  58k
hls-230 mp4 640x360        |             230k m3u8_n | unknown  230k unknown     0k
hls-252 mp4 640x360        |             252k m3u8_n | unknown  252k unknown     0k
mp4-199 mp4 640x360    29  | ~2.37MiB    199k http   | avc1     199k unknown     0k isom
mp4-218 mp4 640x360    29  | ~2.60MiB    218k http   | avc1     218k unknown     0k isom
mp4-220 mp4 640x360    29  | ~2.63MiB    220k http   | avc1     220k unknown     0k isom
hls-433 mp4 1280x720       |             433k m3u8_n | unknown  433k unknown     0k
mp4-386 mp4 1280x720   29  | ~4.60MiB    386k http   | avc1     386k unknown     0k isom
mp4-393 mp4 1280x720   29  | ~4.68MiB    393k http   | avc1     393k unknown     0k isom
mp4-409 mp4 1280x720   29  | ~4.87MiB    409k http   | avc1     409k unknown     0k mp42
[download] Downloading video 2 of 74
[safari:api]<PHONE_NUMBER>012/video301825: Downloading part JSON
[safari]<PHONE_NUMBER>012-video301825: Downloading webpage

@Ashish0804 that's true, I tested a bit, it seems adding the cookies file causes this 404 error, but I don't get how that could be possible when it's available for the public anyway

Try getting latest cookies from the browser (they could have expired)
Also make sure u are able to view/play the files when logged in...as u mentioned u have used this before so depending on how much u were downloading, they could have blocked your account.
If the problem still isn't solved, then you will have to provide the account since i can't reproduce the issue.

You are right, my browser seems to have cached something that lets me access the content still, but when trying to log in from another browser this wasn't the case, I'll solve my account's issues then, thanks.

@Ashish0804 After re-checking with multiple others, I can confirm the issue is from yt-dlp, I can access the site normally and open all videos (The same for my friends), but can't use cookies anymore.
As for the Oreilly account, you can create any temp account and use the free trial.

@pukkandan Did creating a new temp account fail somehow?

The extractor builds the wrong individual video link
This is what the extractor builds:
https://learning.oreilly.com/library/view/getting-started-with/9781787285491/video1_1.html

This is what it should be
https://learning.oreilly.com/videos/getting-started-with/9781787285491/9781787285491-video1_1/

@hasantayyar Thanks, I submitted a PR to handle this change here.
As I didn't read the rest of the code I can't confirm if it creates other issues, but now I can download again without issues.

After some testing, this just bypassed the 404-issue, even after using the right URL, it only downloads 3min from each video as if not authenticated

`[debug] Custom config file: D:\Utils/YoutubeDL/Configs/ytdlp_oreilly.conf
[debug] Custom config: ['--cookies-from-browser', 'firefox', '--download-archive', 'archive.txt', '-i', '-c', '-v', '--console-title', '--batch-file=batch-file.txt', '--write-annotations', '--write-description', '--write-info-json', '--write-thumbnail', '--sub-lang', 'en', '--write-auto-sub', '--write-sub', '--add-metadata', '--embed-subs', '--embed-thumbnail', '-o', '%(playlist_title)s/%(playlist_index)s. %(title)s.%(ext)s', '-f', 'bestvideo[height<=720]+bestaudio/best[height<=720]/worst', '--merge-output-format', 'mp4']
[debug] Command-line config: ['--config-location', 'D:\Utils/YoutubeDL/Configs/ytdlp_oreilly.conf', '--playlist-items', '2']
[debug] Batch file urls: ['https://learning.oreilly.com/videos/the-principles-of/9781491935811/']
[Cookies] Extracting cookies from firefox
[debug] Extracting cookies from: "C:\Users\mh-sh\AppData\Roaming\Mozilla\Firefox\Profiles\8tceuger.default-release\cookies.sqlite"
[Cookies] Extracted 1548 cookies from firefox
[debug] Loading archive file 'archive.txt'
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.09.02 (source)
[debug] Plugin Extractors: ['SamplePlugin']
[debug] Git HEAD: 982323fe1
[debug] Python version 3.9.1 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg n4.4-4-gacb339bb88, ffprobe n4.4-4-gacb339bb88
[debug] Optional libraries: sqlite
[debug] Proxy map: {}
[debug] [safari:course] Extracting URL: https://learning.oreilly.com/videos/the-principles-of/9781491935811/
[safari:course]<PHONE_NUMBER>811: Downloading course JSON
[download] Downloading playlist: The Principles of Microservices
[info] Writing playlist metadata as JSON to: The Principles of Microservices\0. The Principles of Microservices.info.json
WARNING: There's no playlist description to write.
[safari:course] playlist The Principles of Microservices: Collected 15 videos; downloading 1 of them
[download] Downloading video 1 of 1
[debug] [safari:api] Extracting URL: https://learning.oreilly.com/api/v1/book/9781491935811/chapter/video221406.html
[safari:api]<PHONE_NUMBER>811/video221406: Downloading part JSON
[debug] [safari] Extracting URL: https://learning.oreilly.com/videos/the-principles-of/9781491935811/9781491935811-video221406/
[debug] [Kaltura] Extracting URL: https://cdnapisec.kaltura.com/html5/html5lib/v2.37.1/mwEmbedFrame.php?wid=_1926081&uiconf_id=29375172&flashvars[referenceId]=9781491935811-video221406
[Kaltura]<PHONE_NUMBER>811-video221406: Downloading webpage
[Kaltura] 0_1i6jb4o4: Downloading video info JSON
[Kaltura] 0_1i6jb4o4: Checking mp4-4468 URL
[Kaltura] 0_1i6jb4o4: Downloading m3u8 information
WARNING: [Kaltura] Ignoring subtitle tracks found in the HLS manifest; if any subtitle tracks are missing, please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Downloading subtitles: en
[info] 0_1i6jb4o4: Downloading 1 format(s): mp4-1966
WARNING: There's no description to write.
WARNING: There are no annotations to write.
[info] Writing video subtitles to: The Principles of Microservices\2. What are Microservices.en.ttml
[debug] Invoking downloader on "http://cdnapi.kaltura.com/api_v3/service/caption_captionasset/action/serve/captionAssetId/0_diyhxwtn"
[download] The Principles of Microservices\2. What are Microservices.en.ttml has already been downloaded
[download] 100% of 15.08KiB
[info] Writing video metadata as JSON to: The Principles of Microservices\2. What are Microservices.info.json
[Kaltura] 0_1i6jb4o4: Downloading thumbnail  ...
[Kaltura] 0_1i6jb4o4: Writing thumbnail to: The Principles of Microservices\2. What are Microservices.jpg
[debug] Invoking downloader on "http://cdnapi.kaltura.com/p/1926081/sp/192608100/playManifest/entryId/0_1i6jb4o4/format/url/protocol/http/flavorId/0_coc8hy0d"
[download] Resuming download at byte 2717283
[download] Destination: The Principles of Microservices\2. What are Microservices.mp4
[download]   7.6% of 35.87MiB at 489.40KiB/s ETA 01:09`
The issue now is that the downloader is invoked on "http://cdnapi.kaltura.com/p/1926081/sp/192608100/playManifest/entryId/0_1i6jb4o4/format/url/protocol/http/flavorId/0_coc8hy0d" which is only 3min, but it already had access to the full video at "https://cdnapisec.kaltura.com/html5/html5lib/v2.37.1/mwEmbedFrame.php?wid=_1926081&uiconf_id=29375172&flashvars[referenceId]=9781491935811-video221406"

@hasantayyar Thanks, I submitted a PR to handle this change here.
As I didn't read the rest of the code I can't confirm if it creates other issues, but now I can download again without issues.

Thanks @MKSherbini I will test with a subscription.
The issue is the O'Reilly api responds the wrong web url and I think your change is the only way to fix it for now until they changed.

I just installed  with python3 -m pip install --upgrade git+https://github.com/MKSherbini/yt-dlp and was able
to download again with a free trial account.
BUT all videos are truncated after 60 seconds.'
Using the same account in their webplayer let me watch beyond the 60s.

@MKSherbini I tested this with my credentials both with individual video page and course page. It's downloading the videos without issues

I am confused by this conversation. Does #990 download the truncated video, or the full video?

pukkandan, I am sorry for the confusion, as far as I can tell now the fix works just fine!
I updated my original comment  https://github.com/yt-dlp/yt-dlp/issues/586#issuecomment-921203603-permalink

With latest update getting this issue.
[safari:course]<PHONE_NUMBER>709: Downloading course JSON [download] Downloading playlist: Getting Started with Kubernetes LiveLessons, 2nd Edition [safari:course] playlist Getting Started with Kubernetes LiveLessons, 2nd Edition: Collected 87 videos; downloading 87 of them [download] Downloading video 1 of 87 [safari:api]<PHONE_NUMBER>709/GSK2_00_00_00: Downloading part JSON ERROR: no suitable InfoExtractor for URL https:/9780136787709-GSK2_00_00_00 [download] Downloading video 2 of 87 [safari:api]<PHONE_NUMBER>709/GSK2_01_01_00: Downloading part JSON ERROR: no suitable InfoExtractor for URL https:/9780136787709-GSK2_01_01_00 [download] Downloading video 3 of 87 [safari:api]<PHONE_NUMBER>709/GSK2_01_01_01: Downloading part JSON ERROR: no suitable InfoExtractor for URL https:/9780136787709-GSK2_01_01_01 [download] Downloading video 4 of 87 [safari:api]<PHONE_NUMBER>709/GSK2_01_01_02: Downloading part JSON ERROR: no suitable InfoExtractor for URL https:/9780136787709-GSK2_01_01_02 [download] Downloading video 5 of 87 [safari:api]<PHONE_NUMBER>709/GSK2_01_01_03: Downloading part JSON

I see what the issue is. Thanks for the catch. Will fix it when Iget on my PC


Javascript implementation of Piranha

This implements PiranhaJS and the details of usage are in the README.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 3 committers have signed the CLA.:white_check_mark: mkr-plse:x: anag004:x: s7rthakYou have signed the CLA already but the status is still pending? Let us recheck it.

Not a full review yet (not sure I can do one for JS code :) ), but: I see GitHub actions for running the tests, yet nothing currently running on this PR on our existing Travis CI checks (see below the PR, all that is running are the tests for Java code). Can we make sure CI is setup for this code? Either using GitHub actions or Travis, I don't care, as long as they run before every PR is landable and on every commit in master.

The GitHub actions are running in my fork of the repo. Maybe they need to be merged to uber/master before they can run on pull requests?

@msridhar any particular style of documentation that needs to be followed?

I am not particular. For me regular comments are fine or you can use JSDoc. If others are more particular they can comment

I am not particular. For me regular comments are fine or you can use JSDoc. If others are more particular they can comment

Thanks. Let us go with regular comments for now then.

Added docs in 2f7bfe9. Please take a look.

Docs are good!  I'll review again after my other comments (unused esprima depth, realpath) are addressed.

recast internally uses esprima as well as other parsers, so esprima is not unused but is installed as a dependency for recast. Changed the alias  e27407d

LGTM!

Thanks @msridhar for your review.

LGTM

Thanks @lazaroclapp for the review.

Thanks @lazaroclapp, @msridhar for your reviews.


[eonline.com] Add support

[X] I've verified and I assure that I'm running youtube-dl 2017.03.06
[X] At least skimmed through README and most notably FAQ and BUGS sections
[X] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[ ] Bug report (encountered problems with youtube-dl)
[X] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

C:\Users\Desktop\youtube-dl>youtube-dl.exe -f best --all-subs http://www.eonline.com/news/833946/julie-bowen-makes-her-tangled-debut-watch-a-sneak-peek-of-tangled-before-ever-after -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-f', 'best', '--all-subs', 'http://www.eonline.com/news/833946/julie-bowen-makes-her-tangled-debut-watch-a-sneak-peek-of-tangled-before-ever-after', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2017.03.06
[debug] Python version 3.4.4 - Windows-10-10.0.14393
[debug] exe versions: ffmpeg N-82833-gedb4f5d, ffprobe N-82833-gedb4f5d, rtmpdump 2.3
[debug] Proxy map: {}
[generic] julie-bowen-makes-her-tangled-debut-watch-a-sneak-peek-of-tangled-before-ever-after: Requesting header
WARNING: Falling back on generic information extractor.
[generic] julie-bowen-makes-her-tangled-debut-watch-a-sneak-peek-of-tangled-before-ever-after: Downloading webpage
[generic] julie-bowen-makes-her-tangled-debut-watch-a-sneak-peek-of-tangled-before-ever-after: Extracting information
[download] Downloading playlist: Get Your First Look at Julie Bowen's Tangled Character
[generic] playlist Get Your First Look at Julie Bowen's Tangled Character: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[ThePlatform] 1_y4ii75qrZg: Downloading SMIL data
[ThePlatform] 1_y4ii75qrZg: Downloading JSON metadata
[debug] Invoking downloader on 'rtmp://cp423311.edgefcs.net/ondemand/'
[download] Destination: Exclusive Look at 'Tangled Before Ever After'-1_y4ii75qrZg.flv
[debug] rtmpdump command line: rtmpdump --verbose -r rtmp://cp423311.edgefcs.net/ondemand/ -o 'Exclusive Look at '"'"'Tangled Before Ever After'"'"'-1_y4ii75qrZg.flv.part' --playpath mp4:transcoded/rtmp/516/703/clip_20170305_tangled_258122_1600k.flv --resume --skip 1
[rtmpdump] RTMPDump v2.3
[rtmpdump] (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
[rtmpdump] DEBUG: Parsing...
[rtmpdump] DEBUG: Parsed protocol: 0
[rtmpdump] DEBUG: Parsed host    : cp423311.edgefcs.net
[rtmpdump] DEBUG: Parsed app     : ondemand
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Protocol : RTMP
[rtmpdump] DEBUG: Hostname : cp423311.edgefcs.net
[rtmpdump] DEBUG: Port     : 1935
[rtmpdump] DEBUG: Playpath : mp4:transcoded/rtmp/516/703/clip_20170305_tangled_258122_1600k.flv
[rtmpdump] DEBUG: tcUrl    : rtmp://cp423311.edgefcs.net:1935/ondemand
[rtmpdump] DEBUG: app      : ondemand
[rtmpdump] DEBUG: live     : no
[rtmpdump] DEBUG: timeout  : 30 sec
[rtmpdump] DEBUG: Setting buffer time to: 36000000ms
[rtmpdump] Connecting ...
[rtmpdump] DEBUG: RTMP_Connect1, ... connected, handshaking
[rtmpdump] DEBUG: HandShake: Type Answer   : 03
[rtmpdump] DEBUG: HandShake: Server Uptime :<PHONE_NUMBER>
[rtmpdump] DEBUG: HandShake: FMS Version   : <IP_ADDRESS>
[rtmpdump] DEBUG: HandShake: Handshaking finished....
[rtmpdump] DEBUG: RTMP_Connect1, handshaked
[rtmpdump] DEBUG: Invoking connect
[rtmpdump] INFO: Connected...
[rtmpdump] DEBUG: HandleServerBW: server BW = 1250000
[rtmpdump] DEBUG: HandleClientBW: client BW = 1250000 2
[rtmpdump] DEBUG: HandleChangeChunkSize, received: chunk size change to 4096
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 242 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:             fmsVer, STRING:  FMS/5,0,7,7054>
[rtmpdump] DEBUG: Property: <Name:       capabilities, NUMBER:  127.00>
[rtmpdump] DEBUG: Property: <Name:               mode, NUMBER:  1.00>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:              level, STRING:  status>
[rtmpdump] DEBUG: Property: <Name:               code, STRING:  NetConnection.Connect.Success>
[rtmpdump] DEBUG: Property: <Name:        description, STRING:  Connection succeeded.>
[rtmpdump] DEBUG: Property: <Name:     objectEncoding, NUMBER:  0.00>
[rtmpdump] DEBUG: Property: <Name:               data, OBJECT>
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:            version, STRING:  5,0,7,7054>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_result>
[rtmpdump] DEBUG: HandleInvoke, received result for method call <connect>
[rtmpdump] DEBUG: sending ctrl. type: 0x0003
[rtmpdump] DEBUG: Invoking createStream
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 21 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <onBWDone>
[rtmpdump] DEBUG: Invoking _checkbw
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 29 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_result>
[rtmpdump] DEBUG: HandleInvoke, received result for method call <createStream>
[rtmpdump] DEBUG: SendPlay, seekTime=0, stopTime=0, sending play: mp4:transcoded/rtmp/516/703/clip_20170305_tangled_258122_1600k.flv
[rtmpdump] DEBUG: Invoking play
[rtmpdump] DEBUG: sending ctrl. type: 0x0003
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 10275 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <_onbwcheck>
[rtmpdump] DEBUG: Invoking _result
[rtmpdump] DEBUG: HandleChangeChunkSize, received: chunk size change to 4096
[rtmpdump] DEBUG: RTMP_ClientPacket, received: invoke 285 bytes
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: NULL
[rtmpdump] DEBUG: (object begin)
[rtmpdump] DEBUG: Property: <Name:              level, STRING:  error>
[rtmpdump] DEBUG: Property: <Name:               code, STRING:  NetStream.Play.StreamNotFound>
[rtmpdump] DEBUG: Property: <Name:        description, STRING:  Failed to play transcoded/rtmp/516/703/clip_20170305_tangled_258122_1600k.flv; stream not found.>
[rtmpdump] DEBUG: Property: <Name:            details, STRING:  transcoded/rtmp/516/703/clip_20170305_tangled_258122_1600k.flv>
[rtmpdump] DEBUG: Property: <Name:           clientid, STRING:  qAATwhWA>
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: (object end)
[rtmpdump] DEBUG: HandleInvoke, server invoking <onStatus>
[rtmpdump] DEBUG: HandleInvoke, onStatus: NetStream.Play.StreamNotFound
[rtmpdump] ERROR: Closing connection: NetStream.Play.StreamNotFound
[rtmpdump] DEBUG: Closing connection.
[rtmpdump] 0 bytes
[rtmpdump] RTMPDump v2.3
[rtmpdump] (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
[rtmpdump] DEBUG: Parsing...
[rtmpdump] DEBUG: Parsed protocol: 0
[rtmpdump] DEBUG: Parsed host    : cp423311.edgefcs.net
[rtmpdump] DEBUG: Parsed app     : ondemand
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Protocol : RTMP
[rtmpdump] DEBUG: Hostname : cp423311.edgefcs.net
[rtmpdump] DEBUG: Port     : 1935
[rtmpdump] DEBUG: Playpath : mp4:transcoded/rtmp/516/703/clip_20170305_tangled_258122_1600k.flv
[rtmpdump] DEBUG: tcUrl    : rtmp://cp423311.edgefcs.net:1935/ondemand
[rtmpdump] DEBUG: app      : ondemand
[rtmpdump] DEBUG: live     : no
[rtmpdump] DEBUG: timeout  : 30 sec
[rtmpdump] DEBUG: Failed to get last keyframe.
[rtmpdump] DEBUG: Closing connection.

ERROR: rtmpdump exited with code 1
  File "__main__.py", line 19, in <module>
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\__init__.py", line 457, in main
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\__init__.py", line 447, in _real_main
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 1883, in download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 772, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 950, in process_ie_result
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 833, in process_ie_result
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 772, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 826, in process_ie_result
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 1527, in process_video_result
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 1791, in process_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 1733, in dl
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\downloader\common.py", line 356, in download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\downloader\rtmp.py", line 202, in real_download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\downloader\common.py", line 162, in report_error
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 589, in report_error
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp06epbgb6\build\youtube_dl\YoutubeDL.py", line 551, in trouble
...
<end of log>

Single video: http://www.eonline.com/news/833946/julie-bowen-makes-her-tangled-debut-watch-a-sneak-peek-of-tangled-before-ever-after

I would love if you can add this site. It has sneak peeks and videos of stuff.

Duplicate https://github.com/rg3/youtube-dl/issues/6198

It's not the same thing. I ask for support only for free Sneak peeks and clips. I don't need full episodes.

IDM can download the video easily, not sure what the problem will be. But that's for clips, not full eps.

This runs on the platform, can we revisit adding support?

I can provide creds to help test this.


COM-311: Refactor theming of Menu

Desktop:
https://github.com/vivid-planet/comet/assets/109900447/b3063077-0e9e-4e56-94b7-1ef8d7fa3986
Mobile:
https://github.com/vivid-planet/comet/assets/109900447/4ca466df-7acd-4b93-a9ba-b0050ac04c2c

The build/lint should be fixed once https://github.com/vivid-planet/comet/pull/1642 is merged and this is rebased on the feature-branch.

@jamesricky please review


Chore: formatting, fix types and missing dev deps

add format script, (and prettier)
add missing dev deps, (@types/babel__generator, @types/node, tslib)
fix types in create-proxy
adds { internal: true } option to created signals

lgtm. I'd approve this now since I have to rework the HMR codegen