Error in recognizing convexity for power(max(difference,0),n)

Describe the bug
Expressions of the type cvx.power(cvx.max(x-y,0),n) are not always recognized as convex for n>=1.
To Reproduce
import cvxpy as cvx
x = cvx.Variable()
y = cvx.Variable()
cvx.power(cvx.max(x-y,0),2)
cvx.power(cvx.max(x-y,0),3)
cvx.power(cvx.max(x-y,0),4)
cvx.power(cvx.max(x-y,0),5)
cvx.power(cvx.max(x-y,0),6)
Expected behavior
The DCP analyzer at https://dcp.stanford.edu/analyzer returns each of these expressions being positive and convex.
Output
However, CVXPY is telling me that only a subset of these expressions are dcp.

cp.power(cp.max(x-y,0),2)
Expression(UNKNOWN, NONNEGATIVE, ())
cp.power(cp.max(x-y,0),3)
Expression(CONVEX, NONNEGATIVE, ())
cp.power(cp.max(x-y,0),4)
Expression(UNKNOWN, NONNEGATIVE, ())
cp.power(cp.max(x-y,0),5)
Expression(CONVEX, NONNEGATIVE, ())
cp.power(cp.max(x-y,0),6)
Expression(CONVEX, NONNEGATIVE, ())

Version

OS: Mac OS 10.15.6
CVXPY Version: 1.1.1 (with Python 3.6.0)

Additional context
Thanks in advance for any help on this!
Oh also - in cvxpy I am getting that cvx.max(x-y,0) is convex/unknown whereas in the dcp analyzer it is saying convex/positive.

So I realize now that I should have been using cvx.maximum instead of max, so maybe that is intended behavior, but it does seem confusing to the user as it is likely an easy mistake to make...

It definitely is confusing, but this is the numpy convention. The behavior is correct for max(...,0).

Ok - thanks for the reply I'll close it out!


[BEAM-10051] Move closed reader check after sentinel.

The closed reader check,  in it's current position prevents the "normal teardown" that the reader expects. This means that readers for instructions that terminate early such as due to splitting stay resident in memory and never close.
In practice this is benign as the buffer would already be closed, but with streaming this memory leak would become noticable.
This PR moves the check to after the sentinel check, and additionally check there for early termination to avoid closing the buffer twice.
Since this largely affects the goroutine local cache, no test changes are possible.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

R: @youngoli


JLL Registration: JuliaBinaryWrappers/Mineos_jll.jl-v1.0.0+0

Autogenerated JLL package registration

Registering JLL package Mineos_jll.jl
Repository: https://github.com/JuliaBinaryWrappers/Mineos_jll.jl
Version: v1.0.0+0

This PR probably contains a bug.  Please, don't merge it before https://github.com/JuliaPackaging/BinaryBuilder.jl/pull/531.


MSBuild 15 Sdk Design

MSBuild will allow third-party-defined and dynamically-delivered extensions of the build process via the new “Sdk” concept. This will extend the experience delivered with RC.2 to include an acquisition process.
Changes to project files
An Sdk can be consumed in two ways: with implicit top and bottom imports at through the Sdk attribute on the Project element:
<Project Sdk="Microsoft.NET.Sdk">
...
</Project>

and through a modified <Import> element with the Sdk attribute:
<Project>
  <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk/1.0.0" />
...
  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk/1.0.0" />
</Project>

The former mechanism is syntactic sugar for the latter, with an Sdk.props import at the top of the project and an Sdk.targets import at its bottom.
User experience
When a user opens an Sdk-using project in an IDE, the IDE will try to evaluate the project. MSBuild will coordinate with the IDE to fetch and integrate all required Sdks. The IDE will probably wish to display a progress bar for any required downloads.
On the command line, a dotnet restore or msbuild.exe /t:Restore invocation will fetch required Sdks before proceeding with the restore process, ensuring that only one user gesture is required to bring the project to a buildable state.
Sdk acquisition
How does MSBuild know how to acquire an Sdk when it needs to? Through a provider model. A caller of the MSBuild APIs can provide an instance of a new interface ISdkResolver which will be used to map the XML from above to files on the local filesystem.
Note: Although the acquisition could differ between builds the resolved SDK would be identical.
struct SdkReference
{
    public string Name;
    public Version? Version;
}

interface ISdkResolver
{
    public Dictionary<SdkReference, Path> Resolve(IEnumerable<SdkReference>);
}

We expect most ISdkResolver implementations to additionally have affordances for reporting progress back to a UI and to log using the MSBuild log interface.
🚧 Exposing logging may require an interface change here.
🚧 Do we need to pass path-to-project, so that something NuGet-like can walk up to find its feed configuration?
MSBuild evaluation changes
MSBuild will collect Sdks needed for a given project early in its evaluation (before pass 1 expands imports). It will then unify Sdks specified within the project with those specified in the (optional) lineup file. This produces a list of Sdks that is then passed to ISdkResolver.Resolve(), producing a lookup table for Sdk imports.
Evaluator.GetCurrentDirectoryForConditionEvaluation should return the Sdk directory if there is an Sdk attribute on an import, so that you can have something like Condition="Exists('Sdk\Sdk.props')" on an Sdk import.
Evaluation pass 1 (properties and imports) then continues as usual. When considering an Import element with an Sdk attribute, the specified Project attribute is treated as a relative path from the base Sdk path looked up in the table generated in the new pre-pass-1 step.
Lineups
After growing past being “small” projects, most repos/solutions will want a simple way to manage the set of Sdks they use--for instance, to unify versions of Sdks available for projects within the repo. This will be accomplished by a lineup file, format TBD, that MSBuild will consider when putting together the list of available Sdks.
🚧 What does this look like on disk?
We expect a few common patterns of use:

Small-scale “demo projects”

Consist of ~1 project.
Do not have a lineup file.
Specify Sdk version for any “non-core” Sdk referenced.

Larger-scale projects

Have many projects in a solution.
Specify a lineup
Do not specify versions in the project files themselves, only in the lineup.

Project Load
We'll need to add to ProjectLoadSettings so a user can opt out of SDK resolution.  They would need to specify both IgnoreMissingImports and a new DoNotResolveSdks if they wanted to open a project without getting errors.  But this would allow minimal property evaluation without resolving anything.
Concrete work items

[ ] Get feedback from VS folks

Especially around UI needs for status updating.

[ ] Expose ISdkResolver definition for prototyping.
[ ] Implement Sdk-aware imports (#1400)
[ ] Implement Sdk-gathering (pass 0.5)
[ ] Augment build-starting methods to accept an ISdkResolver

In BuildParameters? BuildRequestData?

[ ] Minimal FilesystemSdkResolver for prototyping and testing

Maybe looks in NuGet cache folder? That'd get us a long way toward seeing what the once-everything-is-downloaded scenario looks like.

[ ] Implement NuGetSdkResolver: https://github.com/NuGet/Home/issues/5220

Straw man implementation idea: write sdk names + package versions to a minimal project file, restore it out of proc, return paths 🚧 magically.

[ ] Use resolvers in various scenarios

[ ] Change MSBuild.exe to use a resolver
[ ] Change dotnet CLI to invoke MSBuild with a resolver
[ ] Change VS to invoke MSBuild with a resolver
[ ] Change VS Code to invoke MSBuild with a resolver
[ ] Change VS Mac to invoke MSBuild with a resolver

🚧 Some of those could be helped by changing the default, if blocking evaluation without progress reporting is ok. Probably is for OmniSharp (out of proc) and CLI (synchronous command line build). UIs probably want advanced progress reporting + cancellability.

How does this play with VS Code, JetBrains Rider, Xamarin, MonoDevelop, ...?

@jviau We're designing a way for a UI to provide to MSBuild some logic that resolves an SDK.  The implication for VS is that it would want to present the user with a progress dialog while it restores packages  during project load.  Our design here means that VS would specify a class to MSBuild that we would create an instance of and call Resolve.  If a user loaded a solution with multiple projects, then Resolve would be called multiple times.  Would this work okay with VS?  Can you review the design here and give any feedback?

@maartenba

How does this play with VS Code, JetBrains Rider, Xamarin, MonoDevelop, ...?

We have items near the bottom to involve VS Code, VS Mac and dotnet CLI to ensure they can pass a resolver to give the user a good experience.
JetBrains Rider and MonoDevelop will need to implement the interface for displaying a progress dialog to the user.  I'm fairly certain they already have a custom MSBuild logger to handle log output like Visual Studio so we're hoping it will be easy for IDEs to make this happen for the SDK resolution.  We can probably pave the way with the ones we ship in VS that they can re-use or take as an example.

@AndyGerlicher What do you think about making this more generic and extensible?  Right now it's specifically for resolving SDKs but do you think we'll ever want to resolve other project assets?  My thought was to have the logic be more centered around resolving assets, one of which would be an SDK.  Something like:
public interface AssetResolver
{
    IEnumerable<IResolvedAsset> Resolve(IEnumerable<IAsset> assets);
}

public enum AssetType
{
    None,
    SDK,
}

public interface IAsset
{
    AssetType AssetType { get; }
    string Name { get; }
    Version Version { get; }
}

public interface IResolvedAsset : IAsset
{
    /// <summary>
    /// The path to the resolved asset.
    /// </summary>
    string Path { get; }
}

Thoughts?

I also looked at logging and it's going to be tricky.  The only thing available during project evaluation is the internal LoggingService so we'd need a new interface to expose to the caller or allow them to throw exceptions.  If we want them to be able to log messages and warnings we'll need to come up with something.

This sounds like a very intriguing and valuable feature!

a lineup file, format TBD

Please consider making this an XML file, to keep it consistent with the current ecosystem.  It would be confusing to introduce differing and inconsistent data formats, especially after moving from XML to JSON and then back to XML again.
The spirit of the request here isn't to favor a particular format, but rather to favor consistency.  I might be alone on this, however.  But to me, exploring a given a solution (VS or otherwise) and seeing two different data files in two different formats when they both simply describe data is not only maddeningly inconsistent, but inefficient as well.  That means you ultimately have two libraries/concerns in your solution that parse/save data, when it could be done with just one.
Of course, if you provide a strictly POCO-based model whereby the end user could describe their data in the format they prefer, then this is a non-issue. 😄  What strikes me about this feature is that the same sort of magic could eventually be used to download/install data-format preferences/serialization components as well -- something that has been discussed in https://github.com/Microsoft/msbuild/issues/613.

@jeffki Is he resolver ever an IDE-specific extension or is the IDE's only role to call into msbuild and show progress?

@jeffkl I am a bit confused by the VS UI interaction. Is VS supposed to implement its own ISdkResolver that performs both progress UI and the sdk resolution? Or are you going to have a VS-agnostic implementation that performs only sdk resolution (then it can be re-used for the command line) and then CPS hooks up to events or something and is responsible for displaying the progress UI?

@jviau Yes Visual Studio's implementation of ISdkResolver would do the resolution and the UI.  We didn't think much of the resolvers could be re-used because Visual Studio might use the NuGet object model since it has NuGet assemblies while dotnet CLI might just run NuGet.exe and pipe it's output.  Does this sound doable for VS?

@maartenba The IDE will new up a Project instance for evaluation properties and items and when it does, MSBuild will call the resolvers to acquire SDKs.  The project cannot be evaluated until the SDK exists on disk.  From the command-line, you'll see a log message that MSBuild is acquiring SDKs but we want the UIs to be able to present some progress to the user.  We were assuming that IDEs are already using custom loggers to get info from MSBuild to present to the user so we wanted to follow that same design for SDK resolvers.

@jeffkl will this be a VS-wide implementation, or is it provided per ProjectCollection, ProjectRootElement, or Project? The design above says MSBuild will construct the object, but I think CPS would prefer to construct it and pass it to MSBuild - can we have that behavior available?

@jeffki so looking at the various responses the IDE would be responsible for ánd logging ánd resolving SDK's? Does that also imply a build server should perform both these tasks?

@jviau We think resolvers would be like loggers so they would have a wide scope like ProjectCollection level or build episode.  I think VS should add the resolver to the project collection it uses so the same one is re-used.
MSBuild will be passed an instance of the resolver and call it to resolve any unresolved SDKs.  So yes CPS would construct it.

@maartenba For command-line based builds, MSBuild will ship with a resolver that logs to the console.  So build servers will not need any resolvers.  The main issue is that UI applications would want to pop up their customer dialogs and may have their own way of acquiring SDKs.

@jeffkl I'm concerned that IDE will be able to handle SDK resolving. That way MSBuild will have a different behaviour depending on environment it was started in. It may lead to subtle differences between builds inside IDE and on a build server.

@shalupov I'll update the design to note that although different resolvers could run, the full closure of what they resolve would be identical.  In the case of an SDK, it's just a NuGet package so when building from the command-line the same NuGet package would be used as the one when building from an IDE.  The differences would be in how the end-user application acquires the package (uses its own copy of NuGet, downloads NuGet, uses just the web protocols to get the file) and how it presents progress (a progress dialog or logging to the console).

Lineups

@AndyGerlicher can you explain how these files are different than using a .props file to share version numbers across all projects in a repo?  For example, in the https://github.com/dotnet/sdk, we have a DependencyVersions.props file that specifies the versions of our dependencies: https://github.com/dotnet/sdk/blob/master/build/DependencyVersions.props.  So to update the whole repo to a new NuGet version, we just change a single version number in a single file.
Or are lineups only for managing Sdk versions and not other versions?

Will this changes allso apply to c++ project?

I'm super curious how this meshes with the concept of a nuget package in general. We, https://github.com/Microsoft/winobjc , just moved to ship our "sdk" as a nuget for folks so I'd be curious what the path looks like for sdk writers to move from a nuget to this new hotness.

From API perspective, you shouldn't be returning a concrete Dictionary<TKey, TValue>, instead, IReadOnlyDictionary<TKey, TValue>.

Lineup file could be a special form of an SDK that resolves to a shared / common targets within the solution directory.
I.e
<Project Sdk="Local">

</Project>

Then in the solution dir, you'd have Local.Targets.
<Project>
 <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk/1.0.0" />
<Import Project="Foo.props" Sdk="Foo/1.0.0" />
<Import Project="Sdk.targets" Sdk="Foo.Sdk/1.0.0" />
<Import Project="Sdk.targets" Sdk="Foo.Sdk/1.0.0" />
 </Project>

Local. Targets - acts as the lineup file, i.e you can manage all versions in one place.
This would also allow specific projects to share a different configuration if necessary, e.g:
<Project Sdk="Another">

</Project>

And then you'd add Another.targets alongside your Local.targets.

@dazinator wouldn't that be exactly the same as the Directory.Build.props/Directory.Build.targets support that's already built-in?

Note on lineups: in addition to version unification, it might be helpful/required to be able to specify what resolver should be used to resolve a specified SDK.

Is it still supposed to keep ability to extend MSBuild process using properties like ResolveReferencesDependsOn? Should I report an issue or is this expected? After reviewing https://github.com/Microsoft/msbuild/issues/1392 I was able to extend build process by overriding target AfterResolveReferences using explicit references:
  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk.Web" />

  <Target Name="AfterResolveReferences">
    <Message Text="It Works!" Importance="high"/>
  </Target>

However any changes in the property group didn't make any difference
<ResolveReferencesDependsOn>$(ResolveReferencesDependsOn);ChangesDoesNoteTakeEffect</ResolveReferencesDependsOn>

@mkarpuk Physically where in the project file did you define that property? The defaults are set in Microsoft.Common.CurrentVersion.targets, which is imported through the final import, so you'd have to do the change after an explicit import, much like you did while overriding the target.
Is there a reason you don't want to inject your property using BeforeTargets? I would write your example as
  <Target Name="EmitMessageBeforeResolveReferences"
          BeforeTargets="ResolveReferences">
    <Message Text="It Works!" Importance="high"/>
  </Target>

And then it could be defined anywhere in the file.

I put ResolveReferencesDependsOn in the beginning of the file. Thank you! I didn't realized that property was defined in targets, not props.
BeforeTargets also works, I have just converted from .xproj to new .csproj and explore new format. Everything was nice and clear except for property-based overrides which made me thinking that I don't understand something. Now you solved this, thanks again!

I'm using the msbuild nuget packages to try and build a vs2017 csproj file as part of a unit test. However the build fails with this message:
Message: Microsoft.Build.Exceptions.InvalidProjectFileException : The SDK 'Microsoft.NET.Sdk' specified could not be found.  C:\Users\darre\Source\Repos\dnnvsprojectsystem\src\DnnVsProjectSystem.BuildTools\DnnVsProjectSystem.BuildTools.csproj

Above is talk of an ISdkResolver. What do I need to provide so that the SDK can be resolved? Do I need to set some additional project or global properties?
If I switch the csproj file back to the old format (i.e explicity targets imports) it works.


Info about the credit card fraud target

Hi there,
Would be nice to provide some info about the credit card target example.
We know it's 30 vector size, but what was the training set and what methods was used for classification?
Cheers.

In the source code it mentions this data set:
https://www.kaggle.com/mlg-ulb/creditcardfraud
However I am confused as that one has only 9 features, transformed as PCA, but in the code:

    """Target definition for a binary classifier built using scikit-learn from the Kaggle fraudulent credit card transaction dataset at https://www.kaggle.com/mlg-ulb/creditcardfraud
    """
    target_name = "creditfraud"
    target_data_type = "tabular"
    target_endpoint = "creditfraud_sklearn_pipeline.pkl"
    target_input_shape = (30,)
    target_output_classes = ["benign", "fraud"]
    target_classifier = "BlackBox"
    sample_input_path = "creditcard.npz"

There are 30 features.

Looking at the pipeline:
Pipeline(steps=[('standardscaler', StandardScaler()),
                ('lgbmclassifier', LGBMClassifier(silent=True))])

There is just standard scaler so I am still confused how that dataset of 9 feature column can produce a 30 feature column.

Totally my fault I was fooled by the Kaggle UI, I can see

31 columns one of which is the class so making it 30.
That is all good.
Closing it down!


"Extract text" Regex functions correctly but "Ignore Text" Regex does not

Describe the bug
When inputting regex into the ignore text field it is does not work properly. However, the same regex in the extract text field functions properly.
Version
<IP_ADDRESS>
Steps to reproduce the behavior:

Go to 'ebay.com'
Any search with the "Render anchor tag content" enabled
Input the CSS selectors in the example below
Input the regex: KEYWORD([^);]+) to "Ignore Text" and then into "Extract Text"

Example: https://changedetection.io/share/HrkElfcSpiEa
Expected behavior
As per the screenshot below and regex101.com here: https://regex101.com/r/tfu4vE/1 the word preceding the regex up to the closing parathesis should be matched.
Screenshots
Here is the above example with the regex in the "ignore text" field:

Here is the above example with the regex in the "Extract text" field:

Desktop (please complete the following information):

OS: Ubuntu 20.04
Browser: Basic fast Plaintext/HTTP Client

Additional context
I have run into a similar issue before where the "ignore text" regex does not seem to work across multiple lines. However, I have not tested if that same regex works in the "extract text".

Hmm two important things tho

Ignore text will ignore the whole line when that regex matches
My understanding is that ()'s in "extract text" filter should return only what's inside the ()

Without the extract text Apple([^);]+) regex I get
      Shop on eBay
      Apple iPhone 12 Pro Max 256GB Gold Unlocked Crack screen & Back PARTS ONLY READ
      Apple iPhone 12 Pro Max - 128GB - Pacific Blue (Unlocked) (CA)
      Gold iphone 12 pro max 256gb unlocked used In Excellent Condition
      Apple iPhone 12 Pro Max 512GB Factory Unlocked Diamond 24k Gold Plated Back
      Apple iPhone 12 Pro 128GB 256GB 512GB Unlocked All Colours Good Condition

With the  extract text Apple([^);]+)  regex
 iPhone 12 Pro Max 256GB Gold Unlocked Crack screen & Back PARTS ONLY READ
      Apple iPhone 12 Pro Max - 128GB - Pacific Blue (Unlocked
 iPhone 12 Pro Max 512GB Factory Unlocked Diamond 24k Gold Plated Back
      Apple iPhone 12 Pro 128GB 256GB 512GB Unlocked All Colours Good Condition

Looks like the first result is matching but continuing over, so maybe the problem is that multiline is enabled
Apple iPhone 12 Pro Max 256GB Gold Unlocked Crack screen & Back PARTS ONLY READ

@b4bbuilder so when I use that regex on the regex tester, it does exactly as I predicted

@b4bbuilder can you explain exactly what the goal of your regex is? what exactly is the extract text regex filter meant todo here?

@dgtlmoon I want to be able to ignore a word and have the link be ignored as well. I do not want to extract any text I am just pointing out that when using the regex for that purpose it works how I would like the "ignore text" to work. In the example below I would like to ignore the term "Apple iPhone XR" and have the regex ignore the whole block. This way anytime an Apple iPhone XR is listed for sale a change detection will not be triggered. Currently, if I simply add "Apple iPhone XR" to "ignore text" it will only ignore the one line and anytime a new "Apple iPhone XR" is listed for sale the link that goes with it will still trigger a change detection.
Before adding "Apple iPhone XR" plus the regex to "ignore text":

After adding "Apple iPhone XR" plus the regex to "ignore text":

Note that your examples above do not include the links which is why the regex is not working as I intend it to.

I want to be able to ignore a word and have the link be ignored as well.

hmm what about turn off the link rendering then?

@dgtlmoon The items I intend to search have 30,000+ listings. Currently in order to find added items I would have to copy and paste the title into the search bar which I would prefer to avoid.

so your goal is

watch ebay for a list of items that match a keyword
if that keyword exists, send the title and the URL/link somewhere (for each item)

right?

@dgtlmoon I do not want to match a specific keyword. I want to use eBay's "search title and description" functionality which will provide results that will not always match a keyword. That said my goal is as follows:
-Search anything on eBay
-Exclude words from causing a change detection
-When an item is listed that does not contain an excluded word, send the item title and link
eBay actually already has this functionality but for whatever reason caps the number of words you can exclude to like twenty when I need to ignore far more.


better Metric names and base unit

Hi,
i would appreciate to
a) have prometheus metric naming conventions and perhaps use the same convention than kube-state-metrics
https://prometheus.io/docs/practices/naming/
kube-state-metrics for example: kube_pod_container_resource_limits_cpu_cores
suggestion:
container_cpu_usage => container_cpu_usage_cores
and the values in cores and not millicores
for memory:
kube-state-metrics for example:  kube_pod_container_resource_requests_memory_bytes
suggestion: container_mem_usage => container_memory_usage_bytes
and the value of memory in bytes
Thanks

It's funny you mention kube-state-metrics since I recently discovered it does indeed satisfy my original use case. With that discovery, I was pondering archiving/deprecated this repo :)
As a compromise, the suggested name changes you mentioned sound good. I'd be happy for a PR if you're up for it.

in kube-state-metrics i don't get the current usage. only the configuration. And i need a tool to get as unprivileged user (without clusterroles) cpu,memory usage.
I'm not the great programmer, but i will give it a try ;-) => PR will need some time.

I'll be glad to make the changes -- I just wanted to put that option out there.
Meanwhile, I was thinking more that changing the metric names would break existing users of this image; however, I see there's only 12 pulls. So, now is the time to make this kind of "breaking" change.

...I started to make the code change, but I'm not sure yet how to go from a Quantity to a plain old float64 value that can be exported by Prometheus.
I took an easier path originally and scaled each to an integer value that was in the same units as kubectl top:
https://github.com/itzg/kube-metrics-exporter/blob/f40d9f2fea0ede1e7dd35724b0878fa77f5ec89e/pkg/collector.go#L54-L55
I'll have to research some more.

FYI: I asked a friend to take a look into this (good developer). He has probably time on Thursday
I assume he will create a PR or answer here.


Implement KAST-to-KORE transformation based on definition.kore

~Blocked on: #186~
Introduces:

kast_to_kore: Naive KAST-to-KORE transformation where each EVar.sort is fixed based on the corresponding KVariable.sort if provided, or otherwise will be SortK{}. Hence in general the transformation only results in a well-typed KORE pattern if all variables in the KAST term are already annotated with sorts. Does not require a kompiled definition.
KompiledKore.kast_to_kore: KAST-to-KORE transformation with sort inference based on sentences in definition.kore.
KompiledKore.add_injections: adds sort injections.

I still want to see this broken up.

No need to bring in Kore-level sort inference yet, and the sort inference algorithm used there is pretty high powered when it probably does not need to be (as evidenced that a much simpler inference algorithm has been working on the original kast_to_kore just fine). Also, it appears that the algorithm there will benefit from having a notion of substitutions, because it shouldn't have to do any rewriting (substitutions are simpler).
Move the munging/unmunging out of KPrint class, and also make the KPrint.kast_to_kore just a stub function that:

Calls the original sort inference at the kast level, then
Calls the new kast_to_kore in kompile, then
Calls the new sort injection pass in kompile.

That alone will be a large change to review.

Overall feedback:

Move KompiledDefinition utilities to Definition class in syntax.py. Consider pulling out Definition and other "outer-like" classes into their own module, similar to how we do for the Kast data-structures.
Make konvert.kast_to_kore a simple function which takes as input a KDefinition (Kast), and a Definition (Kore).
Pull out the Kore sort inference, and the Scope class.
No need for the extra test harness yet, just adapt it to use the underlying konvert.kast_to_kore instead of a KPrint.
Wire KPrint to use this method directly, then fallback to kast binary.

The idea is that all Kore data-structure pure things should be in kore/ module, and all Kast should be in kast/ module. The only file which should mention both will be konvert.py.

If you do decide to factor out an outer-like module for Kore, that probably deserves it's own PR.

Move most of the KompiledDefinition utilities to Definition class in syntax.py.

My problem with merging KompiledKore into Definition is that KompiledKore relies on the particular encoding of the subsort relation that the frontend produces, and this change gives the false impression that the methods work on an arbitrary Definiton instance. So in this case I think wrapping a Definition read from disk is more appropriate.

Well, we are really only trying to support definitions that come from the frontend for the moment.... So I guess that's OK.
But we can also change the way we read in subsorts later if that's needed, right? Is the problem that you don't like relying on the attribute subsort? Could instead look for injections in the definition I suppose. But I think it's another case of "if it's working (the subsort attribute), then assume it will keep working until it doesn't, then we make a test for it".
But if you'd like to keep this functionality separate, that's fine. I would think it still belongs in the kore/ subdirectory though, because it has to do with Kore definitions from disk specifically. Then konvert.py can just be about converting between the two formats (and be passed in all the information it needs to be able to do that).

Is the problem that you don't like relying on the attribute subsort?

Basically yes, in particular I don't like that kompile attaches it to an axiom like
axiom{R} \exists{R} (Val:SortKItem{}, \equals{SortKItem{}, R} (Val:SortKItem{}, inj{SortString{}, SortKItem{}} (From:SortString{}))) [subsort{SortString{}, SortKItem{}}()]

which, when instantiating a Definition manually, you'd need to reproduce just so that pyk can read the subsort attribute and build the sort lattice. (Alternatively, you can add dummy axioms instead, but that's not better.) So I figured, let's restrict type inference to definitions read from disk to avoid this, an emphasize it with a wrapper class.
I guess the situation would be better (i.e. less noisy) if kompile had a supersorts or subsorts attribute attached directly to the sort sentence. But that on the other hand would break modularization I guess, e.g. module INT would look different based on the modules it is imported from.

I would think it still belongs in the kore/ subdirectory though

I'm fine with that.

@ehildenb I addressed the review comments above.

Dropped KORE sort inference (from this PR, see below)
Extracted part of KPrint.kast_to_kore to pyk.konvert.kast_to_kore
Moved methods not dealing with the subsort table from KompiledKore to Definition
Moved KompiledKore to pyk.kore.kompiled
Changed TestKoreToKast to use pyk.konvert instead of KPrint

Moreover, I pulled out KORE sort inference for further discussion: https://github.com/runtimeverification/pyk/pull/189. The PR addresses the following point.

Note that right now this kore sort variable inference is a more expensive algorithm than the kast sort variable inference we're doing, and almost certainly is actually not doing any more sort inference than the original algorithm (or else the tests would be failing). So we're doing more work for nothing.

The first commit adds a simple version of KORE sort inference - comparable to KAST sort inference as implemented - that induces a variant of KAST-to-KORE which does not rely on KDefinition while still passes all tests.
The second commit adds a test case that breaks both variants of KAST-to-KORE; then replaces the simple sort inference algorithm with the conceptually (but not computationally) more complex one, which fixes the failing test.

I love how this looks now! Left a few comments.


Bug: The data directory format validation failed

error log:
2023-05-30T07:11:18.731504Z  INFO risingwave_meta::manager::system_param: system parameters: SystemParams { barrier_interval_ms: Some(1000), checkpoint_frequency: Some(10), sstable_size_mb: Some(256), block_size_kb: Some(64), bloom_false_positive: Some(0.001), state_store: Some("hummock+s3://test-useast1-mgmt-bucket"), data_directory: Some("test-useast1-eks-a-rwc-g1h1ln932nf9b8nbd83e14f7a5-jianwei-test"), backup_storage_url: Some("memory"), backup_storage_directory: Some("backup"), telemetry_enabled: Some(true) }
thread 'risingwave-main' panicked at 'Unable to start leader services: SystemParams error: The data directory "test-useast1-eks-a-rwc-g1h1ln932nf9b8nbd83e14f7a5-jianwei-test" is misconfigured.
            Please use a combination of uppercase and lowercase letters and numbers, i.e. [a-z, A-Z, 0-9].
            The string cannot start or end with '/', and consecutive '/' are not allowed.
            The data directory cannot be empty and its length should not exceed 800 characters.

Please confirm the validation specification. cc @wcy-fdu


toList

It would be really convenient to have a toList : Matrix a -> List (List a) function. One potential use case is rendering a matrix that is part of your model.
model.matrix
    |> Matrix.map viewEntry
    |> Matrix.toList
    |> List.map (Html.div rowAttribute)
    |> Html.div matrixAttributes

(Of course, you might try to use Matrix.Extra.prettyPrint in this particular case, but it is too opinionated to be useful in practice.)

Thought about it some more this morning. Maybe there should be two functions, toListOfLists : Matrix a -> List (List a) and toList : Matrix a -> List a.

I think it should just be toList : Matrix a -> List (List a) as you originally suggested. It's trivial to use List.concat on it after :)

Do you want a PR or would you prefer to do it yourself?

Please make a PR!


removing extensions that were replaced by app functionality

Removing extensions that have been replaced by app functionality.

−224,132 nice contribution :)
are all deployed marketplace extensions/apps staying live?

@suevalov and ~195500 lines of the 224000 removed lines came from package-lock.json files 😳

are all deployed marketplace extensions/apps staying live?

My guess is yes since I doubt there is a process to "unbuild" them automatically. I would also assume they will remain live at the last version they were deployed at.
I have kept a few apps which I know are potentially not migrated yet or are being used by 3rd parties.

Are all extensions removed, not being used by 3rd parties?


Automated serial console connections, Cluster prefix idempotence, General shell script linting and documentation

Most of this pull request is general linting and documentation that I've made to my fork to better understand the vm cluster build process. The pull request also includes some key functionality changes to automate the serial console connections and retain idempotence in prefixed-cluster builds.
The linting changes are not crucial to be merged, but I think they help to make the shell scripts a little more understandable.
After these changes, I was able to automatically build a vm cluster with a prefix in 3hrs 15mins.
Waiting for the hosts to install took 9 checks before proceeding.

Automates the serial console connections to the bcpc-vms in tests/automated_install.sh.
These changes are mainly in lines 120-140 of tests/automated_install.sh. The previous screen session either approach failed to connect to the serial consoles or failed to kill them properly.
The main difference here is the use of the screen command argument, bash -c "$WAIT_FOR_HOSTS && $KILL_SCREENS".  This feature builds on the proposed changes of https://github.com/bloomberg/chef-bach/pull/868

Retains idempotence in prefixed-cluster builds
After changing the format of cluster.txt to include a node index as the first field, some of the previous installation scripts break idempotence. Lines 74-77 in vbox_create.sh and line 74 in vm_to_cluster.rb now check that the hostname starts with the prefix. Previous versions would always prepend a non empty prefix to the cluster.txt file.

General shell scirpt linting and documentation
I tried to refactor some of the shell script segments in a simpler way while retaining the original functionality. The pull request makes it look more complicated than it is. It may be a good idea to pull down the pull request branch and actually look at the scripts, mainly tests/automated_install.sh and vbox_create.sh. I realize that changes that make the scripts clearer and simpler to me may not necessarily do the same for others.

Going to close this PR in favor of resolving #1228 with a different approach.
These changes should work for anyone who still wants to take the screened serial consoles approach in a private setting.


Handle Function in redux-actions/handleActions

handleActions in redux-actions can handle anything with toString() method as a action type (https://github.com/acdlite/redux-actions/blob/master/src/handleAction.js#L11). In fact, actions created using createAction has toString() defined to return action type (https://github.com/acdlite/redux-actions/blob/master/src/createAction.js#L39).
I extended type to string | Function, because these two are the most common (either action type or action itself).

But Flow still throws error "Computed property cannot be assigned with function type". How did you get around this?

I've the same "Computed property cannot be assigned with function type" issue. A workaround is to call toString directly on the function.
const noop = createAction('INCREMENT');

const reducer = handleActions({
  [noop.toString()]: (state, action) => ({
    counter: state.counter + action.payload
  })
}, { counter: 0 });

I also still throws error object literal. Computed property cannot be assigned with function type
, after apply patch.
Is there any idea to resolve this?

@tricoder42 I can't merge this in while the specs are failing and there's disagreement on the correctness of the fix. If you're still interested in getting this merged, please rebase and fix the broken specs.


导入数据报错

(zipline_env) ❯ python mongodb.py
Traceback (most recent call last):
  File "mongodb.py", line 305, in <module>
    l.insert_zipline_treasure_format()
  File "mongodb.py", line 240, in insert_zipline_treasure_format
    self.treasure['treasure'].drop()
AttributeError: LoadDataCVS instance has no attribute 'treasure'

@ohblue  mongodb.py文件里面conn函数在main被注释了。你打开去掉main部分注释就好了。

RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
Traceback (most recent call last):
File "/home/liujinhao/zipline-chinese/zipline/data/mongodb.py", line 301, in
l.insert_zipline_treasure_format()
File "/home/liujinhao/zipline-chinese/zipline/data/mongodb.py", line 240, in insert_zipline_treasure_format
self.treasure['treasure'].drop()
AttributeError: 'LoadDataCVS' object has no attribute 'treasure'
去掉了也不行


Update index.md

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sample1/index.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:warning: Validation status: warnings

File
Status
Preview URL
Details

sample1/index.md
:warning:Warning

Details

sample1/index.md

Line 2: [Warning] Invalid file link:(~/image/ATA_config_icon.JPG).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


requirements.txt

requirements.txt it does not have the requests library, which the script needs so much lol

requirements.txt
XD fixed file


Add CFLAGS and instruct nokogiri to use installer libraries.

Fixes mitchellh/vagrant#8594

Wow, that was quick.  You da man! @chrisroberts

@chrisroberts Thanks!
Looking forward to a new Vagrant release with this fix included.
Because currently our plugin vagrant-parallels is incompatible with Vagrant 1.9.5

Can we please have a release that includes this fix?  The vagrant-vsphere plugin is also affected by this bug.

@ggascoigne Yep, I am wrapping up some fixes on the Windows builds and once they are done a new release will be going out.


more detailed instructions

Hi.
The plugin doesn't work for me.
There is Note in Readme

NOTE: In order for pyright to resolve an import, module containing it should be used in the workspace; otherwise import won't be found. Hence the auto-import feature works well in modules loaded with pre-loaded imports.

Can you give more detailed instructions how to use it? Or give link to some guides?
P.S. Pyright imports works from autocompletion menu, so somehow pyright works with imports
NVIM v0.10.0-dev-1816+g2d33a766a
OS: Windows 10 22H2

Hi, can you provide a minimal reproducible example?

Can you give more detailed instructions how to use it? Or give link to some guides?

There's more info on https://github.com/microsoft/pyright/issues/967#issuecomment-679417083, I can add that to the docs.
On the usage side, like noted it works well if you have a symbol you're trying to automatically import that's already imported somewhere else in the project workspace.

Pyright confugured with pyrightconfig.json

I tried with the math module. Plugin message: "no unresolved import error"

math.sqrt()

or select from sqlaclhemy.sql
Plugin message: "no import found for select"
but with authocomplete i can import it

@russarr that's strange, if the pyright provides an "Auto-import" option then the plugin should be able to resolve the import. Can you please post pyright version and nvim version you're using?


CSS Tweak:  Multi-level Ordered Lists

I can't find any examples of multi-level ordered lists in the PHB, so this isn't a big deal, but I think it would make sense to tweak the phb stylesheet to change the list-style for at least three levels of ordered lists, so we get Numbers, Upper-case Roman Numerals, then Lower-case Roman Numerals.  Here is the CSS:
.page ol li ol li {
	list-style:upper-roman;
}

.page ol li ol li ol li {
	list-style:lower-roman;
}

https://github.com/naturalcrit/homebrewery/blob/a71dca1487d2b5e4a9064a19e7e586ed135d67ae/themes/5ePhb.style.less#L83

I think I may have seen nested lists in one of the other sourcebooks. If anyone can track down an example we can try to follow that.

So I asked on the Discord of Many Things server if anyone knew of any examples from 5e sourcebooks, and I quickly got an example that I wasn't expecting:  The equipment lists from the classes.  They look like this:

That's not really what I had in mind, though it technically fits the definition of "list within a list".  Obviously, it's not hard for a user to do this manually so I don't think we really need to pursue adopting this, but just for giggles:
# Equipment
-
	- Sword
	- Shield
	- Helm
-
	- Bow
	- Crossbow
	- Tunic

<style>
.page ul ul {
	display:flex;
}

.page ul ul li {
	list-style:lower-alpha;
	padding-left:0px;
	margin-right:16px
}

</style>

I think this can be closed out and left to individual users if they so desire.  Especially since v3 has been out for awhile and any css change would impact existing brews.  I did put another request on the Discord of Many Things for any eamples in the source books of nested lists so if a good one comes up, we can re-open.


Extend unify/union to handle multiple graphs

This would be an easy task if we were to decide to go through with #20


It appears the opengpg key has expired.

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
╷
│ Error: Failed to install provider
│
│ Error while installing mrolla/circleci v0.6.1: error checking signature:
│ openpgp: key expired
╵

Operation failed: failed running terraform init (exit 1)

Wonder if this is due to the terraform version.

Yes, it was solved with a recent TF version. Sorry for the noise.


Custom Database Path Issue

Hi there,
I'm trying to follow the instructions on http://getcockpit.com/docs#!getting_started/security to move the database storage to another location, but Cockpit doesn't seem to be honouring the setting.
The documentation suggests that this file needs to be created at ./custom/config.php, but we only have that file in ./admin/custom/config.php. Furthermore, if I create the former, and try to run Cockpit, there is no evidence that the file is executed. I tested by putting a call to die("message here"), which was not executed, unless I placed it in ./admin/custom/config.php.
Is it at all possible to move the sqlite files to another path, and if so, what exactly do I need to do?
Thanks!

You need to write in the
cockpit\custom\config.php --> the following code:
 'ca',
	'app.name'     => '_' . COCKPIT_CLIENT,
	'database'	=> [ "server" => "mongolite://".(COCKPIT_DIR. "/" . COCKPIT_FOLDER_STORAGE . "/data"), "options" => ["db" => "cockpitdb"] ],
	'paths'        => [
                '#root'   => COCKPIT_DIR,
                'site'    => COCKPIT_DIR. "/". COCKPIT_CLIENT,
                'storage' => COCKPIT_DIR. "/" .COCKPIT_FOLDER_STORAGE.'',
                '#backups'=> COCKPIT_DIR. "/" .COCKPIT_FOLDER_STORAGE.'/backups',
                'data'    => COCKPIT_DIR. "/" .COCKPIT_FOLDER_STORAGE.'/data',
                'cache'   => COCKPIT_DIR. "/" .COCKPIT_FOLDER_STORAGE.'/cache',
                'tmp'     => COCKPIT_DIR. "/" .COCKPIT_FOLDER_STORAGE.'/cache/tmp',
                'modules' => COCKPIT_DIR.'/modules',
                'assets'  => COCKPIT_DIR.'/assets',
                'custom'  => COCKPIT_DIR.'/../custom',
                'site'    => COCKPIT_DIR == COCKPIT_DOCS_ROOT ? COCKPIT_DIR : dirname(COCKPIT_DIR)
            ],

	'debug'		=> true // enable debug mode (show errors etc.)
	], file_exists(COCKPIT_CONFIG_CLIENT_PATH) ? include(COCKPIT_CONFIG_CLIENT_PATH) : []);

_____________________________

And copy to newFolderStorage/ the current storage contents

Hi there,
Thanks for your help, I set just the database and the paths['data'] keys and that seems to have worked.


if twitch chat auth first with client/secret, pubsub fails to work

I'm guessing here. Need to investigate.  Using a valid custom bot token for chat works fine.

fixed maybe in 4.0.0-rc4?

OK, definitely seeing a case where just chat is configured and the websocket just closes even though auth worked.  This one might be fixed by #662

RC7 has a bunch of fixes related to these problems so hopefully the issues are solved.

rc8 i think finished the deal. seems to be pretty solid now.


MINOR: refine DeleteOffsetsConsumerGroupCommandIntegrationTest#produceRecord

Use try-with-resource to create producer and make sure the producer send record without error.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@FrankYang0529 We reuse the TOPIC and GROUP in the for-loop, and that could be a issue if the cleanup of topic/group is not finished before running next loop. Could you please use different topic/group for each loop?


Add categories to the list of needs

So for example, needs of cloths get grouped together while madrases is grouped with pillows and sleeping bags.

Now that is nifty. I can also see a group being more time oriented, as in "These are the things that we need right now" or even "Next week we will collect stuff at Kungstorget, here is what we need"

At todays meeting we discussed some kind of labeling system. Maybe that is enough. I am going to talk to organizations and see how many needs they would like to list on the page. That would give an indication as to how complex our grouping needs to be.

PR #18 has some very basic grouping, but it's very rigid. Needs are limited to only one category and only one level of categorization is supported. It's a start, but it might not be what we want in the long run. I am leaving this one open for further discussion on the matter


Fix start index under batched_forward_pass

Start index under batched_forward_pass starts from the last query token which does not align with the comment in line 1032. This also causes a problem when handling the response tokens as stated in #1781.
Set start to the first response token instead of the last query token.

@mertsayar8 btw we are now recommending to use the PPOv2Trainer :)

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey, thank you for the PR, sorry for the late reply.
I'm not sure about this. Let's consider the simple following example:
The input is [1, 2, 3, 4, 5, 6, 7], and you predict  [2, 3, 4, 5, 6, 7, 8]. So you want a mask like `[0, 0, 1, 1, 1, 1, 0]
import torch

query_batch = [torch.tensor([1, 2, 3])]
response_batch = [torch.tensor([4, 5, 6, 7])]
attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1, 1]])

# code from PPOTrainer
masks = torch.zeros_like(attention_mask)
masks[:, :-1] = attention_mask[:, 1:]

for j in range(len(query_batch)):
    # ...
    start = len(query_batch[j])  # logprobs starts from the second query token
    if attention_mask[j, 0] == 0:  # offset left padding
        start += attention_mask[j, :].nonzero()[0]
    end = start + len(response_batch[j])

masks[j, :start] = 0
masks[j, end:] = 0

print(masks)  # tensor([[0, 0, 0, 1, 1, 1, 0]])

While we rather expect a mask tensor([[0, 0, 1, 1, 1, 1, 0]])
Am I missing something?


can't install on jupyter 4.1 system python debian sid

No matter what I try the nbextensions folder is never found. (404)
I installed everytihing using PIP.
I also tried to remove all the ipytoh/jupyter related files, anf then a clean installation of just the jupter notebook ... and in the js log I was still having thew (404) error for a missing nbextensions directory.
Can yuo help to debug this?
https://gist.github.com/4fac50855c12ada0a6983ad2d8ff4ec1

maybe useful to debug, when i try to run the single user jupyter notebook:
the link: http://myhost:9000/nbextensions in the js console gives:
require.js?v=6da8be3…:900 TypeError: $ is not a function(…)Module.check @ require.js?v=6da8be3…:900
require.js?v=6da8be3…:900 TypeError: Cannot read property 'extend' of undefined(…)Module.check @ require.js?v=6da8be3…:900
require.js?v=6da8be3…:900 TypeError: $ is not a function(…)Module.check @ require.js?v=6da8be3…:900
require.js?v=6da8be3…:900 TypeError: Cannot read property 'build_page' of undefined(…)

and in the bash shell from where the process is running, i have:
404 GET /static/components/underscore/underscore-min.map (<IP_ADDRESS>) 28.79ms referer=None

In the browser a blank page with entitled Notebook Extension Config is displayed.
The code below:
import os
import glob

from jupyter_core.paths import jupyter_config_dir, jupyter_data_dir

pats = {
    'config': os.path.join(jupyter_config_dir(), '*'),
    'extensions': os.path.join(jupyter_data_dir(), 'extensions', '*'),
    'nbextensions': os.path.join(jupyter_data_dir(), 'nbextensions', '*'),
}
for name, pat in pats.items():
    print('{}:\n  {}'.format(name, '\n  '.join(glob.glob(pat))))

gives me:
config:
  /home/epinux/.jupyter/jupyter_nbconvert_config.py
  /home/epinux/.jupyter/migrated
  /home/epinux/.jupyter/jupyter_nbconvert_config.json
  /home/epinux/.jupyter/jupyter_notebook_config.py
  /home/epinux/.jupyter/nbconfig
  /home/epinux/.jupyter/jupyter_notebook_config.json
nbextensions:
  /home/epinux/.local/share/jupyter/nbextensions/codemirrormode
  /home/epinux/.local/share/jupyter/nbextensions/gist.js
  /home/epinux/.local/share/jupyter/nbextensions/config
  /home/epinux/.local/share/jupyter/nbextensions/jupyter-js-widgets
  /home/epinux/.local/share/jupyter/nbextensions/jupyter-leaflet
  /home/epinux/.local/share/jupyter/nbextensions/usability
  /home/epinux/.local/share/jupyter/nbextensions/ipyparallel
  /home/epinux/.local/share/jupyter/nbextensions/styling
  /home/epinux/.local/share/jupyter/nbextensions/slidemode
  /home/epinux/.local/share/jupyter/nbextensions/testing
  /home/epinux/.local/share/jupyter/nbextensions/publishing
extensions:
  /home/epinux/.local/share/jupyter/extensions/pre_pymarkdown.py
  /home/epinux/.local/share/jupyter/extensions/pre_svg2pdf.py
  /home/epinux/.local/share/jupyter/extensions/pre_codefolding.py
  /home/epinux/.local/share/jupyter/extensions/highlight_html_cfg.py
  /home/epinux/.local/share/jupyter/extensions/strip_output_prompt.py
  /home/epinux/.local/share/jupyter/extensions/highlight_latex_cfg.py
  /home/epinux/.local/share/jupyter/extensions/pre_collapsible_headings.py
  /home/epinux/.local/share/jupyter/extensions/nbextensions.py
  /home/epinux/.local/share/jupyter/extensions/__pycache__
  /home/epinux/.local/share/jupyter/extensions/pp_highlighter.py
  /home/epinux/.local/share/jupyter/extensions/post_embedhtml.py

/home/epinux/.jupyter/nbconfig/ edit.json

{
  "load_extensions": {
    "jupyter_cms/editor/main": false
  }
}

/home/epinux/.jupyter/nbconfig/ notebook.json
{
  "load_extensions": {
    "codefolding": true,
    "jupyter-leaflet/extension": true,
    "jupyter_cms/notebook/main": false,
    "jupyter-js-widgets/extension": true
  }
}

/home/epinux/.jupyter/nbconfig/ tree.json
{
  "load_extensions": {
    "jupyter_cms/dashboard/main": false,
    "ipyparallel/main": true
  }
}

/home/epinux/.jupyter/nbconfig/ usability.json
{
  "load_extensions": {
    "codefolding": true
  }
}

/home/epinux/.jupyter/ / jupyter_nbconvert_config.json
{
  "Exporter": {
    "preprocessors": [
      "pre_codefolding.CodeFoldingPreprocessor",
      "pre_pymarkdown.PyMarkdownPreprocessor"
    ],
    "template_path": [
      ".",
      "/home/epinux/.local/share//jupyter/templates"
    ]
  },
  "NbConvertApp": {
    "postprocessor_class": "post_embedhtml.EmbedPostProcessor"
  },
  "version": 1
}

/home/epinux/.jupyter/ / jupyter_nbconvert_config.py
#--- nbextensions configuration ---
import sys
sys.path.append('/home/epinux/.local/share/jupyter/extensions')
#--- nbextensions configuration ---

/home/epinux/.jupyter/ / jupyter_notebook_config.json
{
  "NotebookApp": {
    "extra_template_paths": [
      "/home/epinux/.local/share/jupyter/templates"
    ],
    "nbserver_extensions": {
      "nbextensions": true
    }
  },
  "version": 1
}

/home/epinux/.jupyter/ / jupyter_notebook_config.py
#--- nbextensions configuration ---
import sys
sys.path.append('/home/epinux/.local/share/jupyter/extensions')
#--- nbextensions configuration ---

/home/epinux/.jupyter/ / migrated
2016-06-04T22:39:39.713878

The nbextension and extension are also found in:  /root/.jupyter/

While this:
import notebook
notebook.nbextensions.check_nbextension('usability/codefolding', user=True)
notebook.nbextensions.check_nbextension('usability/codefolding/main.js', user=True)

returns True as expected.
This:
import notebook
E = notebook.nbextensions.EnableNBExtensionApp()
E.enable_nbextension('usability/codefolding/main')

Returns:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-79dd84cd75e5> in <module>()
      1 import notebook
      2 E = notebook.nbextensions.EnableNBExtensionApp()
----> 3 E.enable_nbextension('usability/codefolding/main')

AttributeError: 'EnableNBExtensionApp' object has no attribute 'enable_nbextension'

After a long debug I think I fixed, most of the isse. I'll clode this and open a new one for the extensions which are not running/loadig on the jupyterhub

apologies for late reply, just seen this. I guess this was partly down to notebook versioning issues with your enable call, and partly down to the fact that our current install scripts only ever put files in jupyter --data-dir...

and what does
import notebook
print(notebook.version_info)

give?

Hi, I just cleaned up all my config, reinstalled python using conda and reinstalled the extension. I was finally able to load the configuration page using jupyter-notebook.
I then installed jupyterhub (my target is to have the extensions working for all the users) .
Running the hub as normal user, I can access to the nbextensions as I do with the jupyter-notebook (single user).
The problem arise when the the hub is started via sudo in that case the nbextensions page resultgs in a blank page.

import notebook
print(notebook.version_info)
(5, 0, 0, '.dev')

ah, ok. So, the fact that you get a blank page and not a 404 implies that the server responds, but doesn't manage to find the appropriate javascript

Ah, 5.0! There be dragons (maybe)

yes! unfortunately I'm using PAM authentication method and sudo is required to run a multi user server

So, currently, all of our javascript gubbins gets installed into the per-user jupyter --data-dir. So, to get round this, you can:

install this repo using sudo, to (hopefully) get the same jupyter --data-dir that the server will search
set environment variable JUPYTER_DATA_DIR before installing, in order to install to a custom location, which you can set to one of the locations in the jupyter path that the server searches

You can list the nbextensions directories which the server will search using
import os.path
from jupyter_core.paths import jupyter_path
for base in jupyter_path():
    print(os.path.join(base, 'nbextensions'))

(since your server is running as super-user, you could run this python script as super-user to get appropriate results, or alternatively, use the system jupyter path defined at jupyter_core/paths.py#L97-L107)

Presumably, you should be able to get js installed into an appropriate location using
JUPYTER_DATA_DIR='/usr/share/jupyter'
pip install https://github.com/ipython-contrib/IPython-notebook-extensions/archive/master.zip

since jupyter_core.paths used in our install script checks the JUPYTER_DATA_DIR envrionment variable. This should all get easier once we switch to a proper pip package (see my master branch for details, if you're interested)

This is what I use in the jupyterhub configuration file:
c = get_config()

os.environ['JUPYTER_DATA_DIR'] = '/usr/local/share/jupyter/'

When I try to install the nbextensions using sudo (or root) it tries to install in /root/.jupyter
root@Debian-70-wheezy-64-minimal:/home/epinux/dev/IPython-notebook-extensions# python setup.py install
Running source install...
Extensions and templates path: /usr/local/share/jupyter/
Installing Jupyter notebook extensions.
Done!
Configuring extensions...
Configuring the Jupyter notebook extensions.
Configuration files directory: /root/.jupyter
Done!
..
...

From the hub running under normal user I have:
/usr/local/share/jupyter/nbextensions
/home/epinux/anaconda3/share/jupyter/nbextensions
/usr/local/share/jupyter/nbextensions
/usr/share/jupyter/nbextensions

and from the Hub running under sudo:
/usr/local/share/jupyter/nbextensions
/usr/local/share/jupyter/nbextensions
/usr/share/jupyter/nbextensions

When running with sudo I got the blank page with the foillowing js log in the console:
require.js?v=6da8be3…:900 TypeError: $ is not a function(…)Module.check @ require.js?v=6da8be3…:900
require.js?v=6da8be3…:900 TypeError: Cannot read property 'extend' of undefined(…)Module.check @ require.js?v=6da8be3…:900
require.js?v=6da8be3…:900 TypeError: $ is not a function(…)Module.check @ require.js?v=6da8be3…:900
require.js?v=6da8be3…:900 TypeError: Cannot read property 'build_page' of undefined(…)Module.check @ require.js?v=6da8be3…:900

when installing with sudo the data goes in the right place, while the config goes in :
 Configuration files directory: /root/.jupyter

Ah, ok, so the console errors about $ mean that this is something
different to what I was thinking.
I thought that the server was just throwing back a 404 for the script
required to load the nbextensions config page, but it seems that actually
it's more complicated than that.
It seems as though something in the page is breaking related to a jquery
requirement.
I have a feeling that this may actually be down to some change in how
jquery is loaded in notebook 5.0 - would it be possible to try running with
4.x?
If you could run the server with the --debug flag, we might see in the
server logs what's getting requested/404'ed, although it seems that may
have been a red herring. I'm going to sleep now, but will try to check this
in the morning...
On 6 Jun 2016 2:58 a.m., "epifanio"<EMAIL_ADDRESS>wrote:

when installing with sudo the data goes in the right place, while the
config goes in :
Configuration files directory: /root/.jupyter
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ipython-contrib/IPython-notebook-extensions/issues/617#issuecomment-223853687,
or mute the thread
https://github.com/notifications/unsubscribe/AFXq9kFQ4CTuEt0S6lx-HI-Y7Y_jXaKWks5qI37dgaJpZM4IuPpI
.

Thanks for the help! I'll try a downgrade and see if changes (but i remember 4.x was giving me trouble as well, I upgraded to 5.0 today. I open this issue running 4.1)
I'm using master brach from your git repo, If you want debug this on a specific branch I can switch to it.
For what I saw I guess the issue is related to the config files not the widgets data, I run the hub without sudo I have access to the configuration page (but can't run multiuser) I enabled some widget for just my user. The: stop the hub, re-run with sudo and the widgets I enabled before are still there and working properly. Is just the configuration-page which doesn't load.
I'll enable debug and post the log (I actually installed the hub with pip install -e. from the git repo, I should be able to open a shell and see the js logs as well.)
I'll post more tomorrow, Thanks!

When I  refersh the browser page, pointing to the NBEXTENSIONS configuration page ( which is rendering a blank page) I saw this log in the jupyter shell outup:
[W 2016-06-06 04:34:19.345 epinux log:47] 404 GET /user/epinux/static/components/underscore/underscore-min.map (<IP_ADDRESS>) 1.91ms referer=None

More from the js console:
require.js:900 TypeError: Cannot read property 'build_page' of undefined
    at http://epinux.com:8888/user/epinux/nbextensions/:204:23
    at Object.context.execCb (http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:1690:33)
    at Object.Module.check (http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:865:51)
    at Object.<anonymous> (http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:1140:34)
    at http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:131:23
    at http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:1190:21
    at each (http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:56:31)
    at Object.Module.emit (http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:1189:17)
    at Object.Module.check (http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:940:30)
    at Object.<anonymous> (http://epinux.com:8888/user/epinux/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce:1140:34)Module.check @ require.js:900

For what I saw I guess the issue is related to the config files not the widgets data

Apologies, yes, from your description (works as regular user, not as super), I think you're right. I was clearly getting a little over-tired 😝
So, there seem to be a few errors relating to missing javascript files, but they all seem to relate to components of notebook, rather than anything from this repo, so I'm a bit mystified.
The server console error is just a sourcemap for the minified version of the underscore library missing from somewhere, so probably not a big deal that it's missing, as the minified version should still be available.
The set of javascript console errors seem to be caused by missing notebook components:

require.js:900 TypeError: Cannot read property 'extend' of undefined
    at http://epinux.com:8888/user/epinux/static/base/js/keyboard.js?v=20160606043728:65:10

seems to result from a missing/undefined jquery in notebook/static/base/js/keyboard.js#L62-64

require.js:900 TypeError: $ is not a function
    at Object.get_body_data (http://epinux.com:8888/user/epinux/static/base/js/utils.js?v=20160606043728:521:19)

seems to be another missing jquery ($), I think from notebook/static/base/js/utils.js#L581.
Finally,

require.js:900 TypeError: Cannot read property 'build_page' of undefined
    at http://epinux.com:8888/user/epinux/nbextensions/:204:23

is just a result of the previous two errors, which mean the nbextensions configurtor-page javascript didn't define a proper module because it threw errors on missing required notebook components (utils and keyboard).
I'm still none the wiser why they're missing a jquery.
Could you try in a (multi-user) notebook window running from a javascript console
require('base/js/utils', function () { console.log('success:', arguments); }, function () { console.log('error:', arguments); });

?

also, from your gist https://gist.github.com/epifanio/4fac50855c12ada0a6983ad2d8ff4ec1/70dd7aeedc8f78f968930dc3188078b8fd9b36cf, it seems that you're running
sudo jupyter nbextension enable codefolding

when actually, you need the full require path to the nbextension entry-point, which would be
sudo jupyter nbextension enable usability/codefolding/main

Ok, seems I find the culprit. Switching between session and users from the same ssh login shell is not as safe as I thought. The environment between the users get confused.
The situation now is:

I can load the extensions for all the users
I can have access to the NBEXTENSIONS configuration page for all the users
The extensions are listed as incompatible, but once activated they works fine
in the  NBEXTENSIONS configuration page the extensions are listed twice

To have this working I installed as sudo the main repository:
git clone https://github.com/ipython-contrib/IPython-notebook-extensions
cd IPython-notebook-extensions
sudo python3 setup.py install

At this point the extensions are on the server but the other users can't access to the NBEXTENSIONS configuration page, it gives a 404 error. To fix this I opened a new terminal and I logged in a new ssh session with the new user and I installed the extensions with:
pip install https://github.com/ipython-contrib/IPython-notebook-extensions/archive/master.zip --user

While this fix the availability of extensions for the user, it will add a bouch of duplicate libraries in the usser's local environment see installation log
then stop the user notebook server. log-out, log-in and now the user has access to the configuration page:

Uncheck the box to enable incompatible extensions:

As you can see all the extensions gets duplicated, I thought because it is finding the extensions system wide and user side (.local/share/jupyter/nbextensions) so I tried to remove the user's extensions running from the user's ssh shell:
rm -rf .local/share/jupyter/nbextensions/publishing
rm -rf .local/share/jupyter/nbextensions/slidemode
rm -rf .local/share/jupyter/nbextensions/styling
rm -rf .local/share/jupyter/nbextensions/testing
rm -rf .local/share/jupyter/nbextensions/usability

But also after stopping the server and logging in again, I still have the extensions listed twice.
I tested some of the extensions and they works (note: I have to check both entries to have them enabled), except for the latex_envs which requires the changes you have in the relative branch.
I tried to apply them, this is the log trying to load the latex_envs . the log points to http://epinux.com:8888/user/epinux/nbextensions/usability/latex_envs/latex_envs.js to which I already applied the changes you suggested (seems they do not work anymore)
For the duplicate etryies, my best guess is that system wide, the extensions get installed in different location and are loaded twice but after:
updatedb
locate latex_envs

I got:
/home/epinux/anaconda3/conda-bld/work/nbextensions/usability/latex_envs/latex_envs.js
/home/epinux/dev/IPython-notebook-extensions/nbextensions/usability/latex_envs/latex_envs.js
/usr/local/share/jupyter/nbextensions/usability/latex_envs/latex_envs.js

from:
import os.path
from jupyter_core.paths import jupyter_path
for base in jupyter_path():
    print(os.path.join(base, 'nbextensions'))

I got:
/usr/local/share/jupyter/nbextensions
/usr/local/share/jupyter/nbextensions
/usr/share/jupyter/nbextensions

Here you can see the system directory is printed twice .. can be a clue?  At this point I think the duplication depends on the configuration files. can it be?
From the main user notebook:
import os
import glob
from jupyter_core.paths import jupyter_config_dir, jupyter_data_dir
ls {jupyter_config_dir()}

returns:
jupyter_nbconvert_config.json  jupyter_notebook_config.json  migrated
jupyter_nbconvert_config.py    jupyter_notebook_config.py    nbconfig/
jupyter_nbconvert.json         jupyter_notebook.json

This is the content of
jupyter_notebook_config.json
{
  "NotebookApp": {
    "extra_template_paths": [
      "/home/epinux/.local/share/jupyter/templates"
    ],
    "nbserver_extensions": {
      "nbextensions": true
    },
    "server_extensions": [
      "nbextensions"
    ]
  },
  "version": 1
}

jupyter_notebook_config.py
#--- nbextensions configuration ---
import sys
sys.path.append('/home/epinux/.local/share/jupyter/extensions')
#--- nbextensions configuration ---
from jupyter_core.paths import jupyter_config_dir, jupyter_data_dir
import os
import sys

sys.path.append(os.path.join(jupyter_data_dir(), 'extensions'))
c = get_config()
c.NotebookApp.extra_template_paths = [os.path.join(jupyter_data_dir(),'templates') ]
c.Exporter.template_path = [os.path.join(jupyter_data_dir(), 'templates') ]

and jupyter_notebook.json
{
  "Exporter": {
    "preprocessors": [
      "pre_codefolding.CodeFoldingPreprocessor",
      "pre_pymarkdown.PyMarkdownPreprocessor"
    ]
  },
  "NbConvertApp": {
    "postprocessor_class": "post_embedhtml.EmbedPostProcessor"
  },
  "NotebookApp": {
    "server_extensions": [
      "nbextensions"
    ]
  },
  "version": 1
}

btw, .. @jcb91  I'm so happy to have them back working and for all the users! thanks a lot for all this work and to taking the time to look into my issue! :)

Ah, you're welcome @epifanio, it's always nice to know that somebody's using the stuff :)
So, I think the duplicates might be an unintended result of my earlier suggestion in https://github.com/ipython-contrib/IPython-notebook-extensions/issues/617#issuecomment-223852006, that of setting the JUPYTER_DATA_DIR variable to be one of the directories from the system path. From

I got:
/usr/local/share/jupyter/nbextensions
/usr/local/share/jupyter/nbextensions
/usr/share/jupyter/nbextensions

We can see that /usr/local/share/jupyter/nbextensions appears twice over, so I think this is where the duplicates are coming from. From

This is what I use in the jupyterhub configuration file:
c = get_config()

os.environ['JUPYTER_DATA_DIR'] = '/usr/local/share/jupyter/'

I guess what's happening is that the environment-variable path gets blindly prepended to the system path, without checking whether it's already in there. So, you should hopefully be able to fix it removing the relevant line in your jupyterhub configuration.

As for each user being able to see the nbextensions configuration page, this relies on the server extension being specified in a relevant jupyter_notebook_config.json, for which we can use a similar trick to setting the JUPYTER_DATA_DIR environment variable to spoof the current install script. I should really have mentioned that when I suggested the previous trick.

A full system-wide install of the repo should be possible using (bash):
JUPYTER_CONFIG_DIR=$(python -c "from jupyter_core.paths import SYSTEM_CONFIG_PATH as p; print(p[0])")
JUPYTER_DATA_DIR=$(python -c "from jupyter_core.paths import SYSTEM_JUPYTER_PATH as p; print(p[0])")
export JUPYTER_CONFIG_DIR JUPYTER_DATA_DIR
pip install https://github.com/ipython-contrib/IPython-notebook-extensions/archive/master.zip
unset JUPYTER_CONFIG_DIR JUPYTER_DATA_DIR

(which of course requires write permissions to the relevant directories , which are /usr/local/etc/jupyter and /usr/local/share/jupyter for unix-based systems)

I upgraded all my installations to latest git master for ipython, notebook, jupyter, jupyterhub, ipywidgets I confirm the nbextensions are working fine for all the users (i'll try to debug latex_envs later on) and the config page loads for all users as well (with the duplicate effect)
@jcb91 what do you suggest? should I just run the commands below without baking-up or removing existent installation?

Well, if it for you now, there's no pressing need to change. The only real difference will be if/when you need to add new users: with the existing install, you'll need to make sure their config files are modified as with those you've already done. Whereas with a system -wide config in place, you wouldn't need to do anything extra...

I ran as root:

JUPYTER_CONFIG_DIR=$(python -c "from jupyter_core.paths import SYSTEM_CONFIG_PATH as p; print(p[0])")
JUPYTER_DATA_DIR=$(python -c "from jupyter_core.paths import SYSTEM_JUPYTER_PATH as p; print(p[0])")
export JUPYTER_CONFIG_DIR JUPYTER_DATA_DIR
pip install https://github.com/ipython-contrib/IPython-notebook-extensions/archive/master.zip
unset JUPYTER_CONFIG_DIR JUPYTER_DATA_DIR

then added a new unix user on the server and use it to login on the hub, I got the same behaviour. 404 at the config nbextension page. Logging in with previous users still has the duplicated entryies in the config page (which is accessible, no-404)

I ran as root:

Could you post any logs form the install - what file(s) got written?

I got the same behaviour. 404 at the config nbextensions page

Hmm. Ok, I'm not exactly sure what's going on there. In the server logs (when run with --debug flag), you should be able to find which config files get loaded - could you check? Do you get any warning logs about being unable to import the extension (module name is  nbextensions)? If not, then it seems that the relevant config files aren't requesting it be loaded...

also, I assume you restart the server after the install?

@jcb91 yes I stopped the hub including the http_proxy and restart it from the shell. p.s. If you think can be  useful I grant you access to the hub. I'm on gitter #ipywidgets #ipython #jupyter

I'll rerun the installation, save the shell output then I'll restart the server and check the logs in both shell and js console.

okey doke 👍

I'm not sure how single-user server logs get handled - do they go to the same place as the hub server logs?

yes, each action on the hub (any user) is notified in the shell. I usually start the server with:
nohup sudo [my-jupyter-start-script.sh] &
so all the logs are stored in the nohup.out file.

Righto, I'll await logs with interest. I've never used gitter, so don't really know where to start with that, but feel free to move conversation there if it's more convenient.

If you think can be useful I can add you as user on the hub

Possibly, if all the kernels are running on the same machine as the hardware I guess I could read relevant config files more quickly? Not sure.

I made a pr to add a gitter badge https://github.com/ipython-contrib/IPython-notebook-extensions/pull/624

👍 merged

Installation logs
HUB restart
root@Debian-70-wheezy-64-minimal:~# sh /usr/local/bin/geopy.sh
root@Debian-70-wheezy-64-minimal:~# nohup: appending output to 'nohup.out'

Output in nohup.out after the following actions:

adduser jcb91
log-in as admin in the hub -> admin control panel -> add user jcb91 -> start jcb91 server -> log out
log-in as jcb91 -> start an empty python3 notebook -> go in the browser address and try to load http:my.hub.domain:8888/user/jcb91/nbextensions

this the nohup.out log: https://gist.github.com/c42ed8cfe84c684621cfc9c39b3172f2  (ignore the error frol L43-L187 was my mistake, I forgot to add the directory used to store the notebooks in the user home)
this the js console while loading the first notebook and then trying to load the nbextensions configuration page: https://gist.github.com/e5f47b42ad90e1fefc3746ac36a478e1

this should now be a lot simpler with our new 2-part installation method:
pip install jupyter_contrib_nbextensions
jupyter contrib nbextension install --system

(as always, writing to system config files and/or jupyter data directories may require root privileges).
So, I'll close this assuming it's fixed. Please open a new issue if the new method doesn't work as intended...


Respect max rowgroup size in Arrow writer

Which issue does this PR close?
Closes #257.
Rationale for this change
Parquet splits batches into row groups, which are normally determined by a max_row_group_size setting.
The Arrow writer could not respect this setting because we cannot slice into structs and arrays correctly.
The issue is that when using array.slice(offset: usize, len: usize), we don't propagate and calculate the slice of child data, leading to only the top-level data being sliced.
What changes are included in this PR?
We use the LevelInfo struct to keep track of its array's offset and length. This allows us to track nested arrays' offsets, and calculate the correct list offsets and lengths.
We then use the arrow::array::slice to perform 0-copy slices from a batch, to limit the row group size correctly.
I have changed all writer tests to use a max row group size, ensuring that we aren't introducing bugs when slicing.
Note that this is related to #225, but I don't think it quite covers all its use-cases.
If we have a sliced recordbatch per #343, we would need to account for its individual array offsets, as there is never a guarantee that a record batch has all child arrays starting from the same offset.
Are there any user-facing changes?
No. All changes are crate-internal.

CC @crepererum @houqp (we've spoken about this before)

Codecov Report

Merging #381 (27b8be6) into master (f26ffb3) will increase coverage by 0.02%.
The diff coverage is 92.17%.

@@            Coverage Diff             @@
##           master     #381      +/-   ##
==========================================
+ Coverage   82.60%   82.62%   +0.02%
==========================================
  Files         162      162
  Lines       44199    44276      +77
==========================================
+ Hits        36509    36584      +75
- Misses       7690     7692       +2

Impacted Files
Coverage Δ

parquet/src/arrow/levels.rs
82.69% <86.79%> (+0.09%)
:arrow_up:

parquet/src/arrow/arrow_writer.rs
98.03% <96.72%> (-0.08%)
:arrow_down:

parquet/src/file/properties.rs
95.74% <100.00%> (+0.01%)
:arrow_up:

arrow/src/array/builder.rs
85.29% <0.00%> (ø)

arrow/src/compute/kernels/concat.rs
100.00% <0.00%> (ø)

arrow/src/compute/kernels/window.rs
100.00% <0.00%> (ø)

parquet/src/arrow/array_reader.rs
77.18% <0.00%> (+0.09%)
:arrow_up:

parquet/src/encodings/encoding.rs
95.04% <0.00%> (+0.19%)
:arrow_up:

arrow/src/array/transform/mod.rs
89.18% <0.00%> (+0.24%)
:arrow_up:

arrow/src/array/array_binary.rs
90.15% <0.00%> (+0.29%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f26ffb3...27b8be6. Read the comment docs.

I could not automatically cherry pick this to active_release (what will become 4.3.0) so if we want to do so we will have to do it manually
alamb@ip-10-0-0-124:~/Software/arrow-rs$ CHERRY_PICK_SHA=7ec3158     ARROW_GITHUB_API_TOKEN=$ARROW_GITHUB_API_TOKEN CHECKOUT_ROOT=/tmp/arrow-rs  python3 dev/release/cherry-pick-pr.py
CHERRY_PICK_SHA=7ec3158     ARROW_GITHUB_API_TOKEN=$ARROW_GITHUB_API_TOKEN CHECKOUT_ROOT=/tmp/arrow-rs  python3 dev/release/cherry-pick-pr.py
Using checkout in /tmp/arrow-rs
Creating cherry pick from 7ec3158b to cherry_pick_7ec3158b
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
From github.com:apache/arrow-rs
 * branch              master     -> FETCH_HEAD
From github.com:apache/arrow-rs
 * branch              active_release -> FETCH_HEAD
fatal: A branch named 'cherry_pick_7ec3158b' already exists.
Command failed: ['git', 'checkout', '-b', 'cherry_pick_7ec3158b']
With output:
--------------
b''
--------------
Traceback (most recent call last):
  File "/Users/alamb/Software/arrow-rs/dev/release/cherry-pick-pr.py", line 153, in <module>
    make_cherry_pick()
  File "/Users/alamb/Software/arrow-rs/dev/release/cherry-pick-pr.py", line 114, in make_cherry_pick
    run_cmd(['git', 'checkout', '-b', new_branch])
  File "/Users/alamb/Software/arrow-rs/dev/release/cherry-pick-pr.py", line 78, in run_cmd
    raise e
  File "/Users/alamb/Software/arrow-rs/dev/release/cherry-pick-pr.py", line 70, in run_cmd
    output = subprocess.check_output(cmd)
  File<EMAIL_ADDRESS>line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File<EMAIL_ADDRESS>line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'checkout', '-b', 'cherry_pick_7ec3158b']' returned non-zero exit status 128.
alamb@ip-10-0-0-124:~/Software/arrow-rs$


SIGSEGV during run

Hi michael,
I just tried your k means (c++) implementation but only got a SIGSEGV :-(
It seems like Eigen is not able to assign the rows correctly during the initial sampling in this line.
My code looks just like this:
Eigen::MatrixXd data = Eigen::MatrixXd::Random(100000, 2);
double* mu = 0;
double* z = 0;
RunKMeans(data.data(), data.rows(), data.cols(),
          10, 15,
          42, "random",
          mu, z);

But the output is just a SIGSEGV with the latest version of Eigen.
Have you seen similar things? Otherwise I will dig into the problem a little bit more...

This isn't an Eigen-related bug... this is simply due to the fact that mu and z in your example here are empty pointers and not valid pointers to arrays of the right size. When you call RunKMeans, it is expected that you hand off preallocated arrays of the right size:
mu : needs to be a pointer to an array of doubles of size K x D=data.cols()
z : needs to be a pointer to an array of doubles of size N=data.rows()
See the newly-committed example here:
https://github.com/michaelchughes/KMeansRex/blob/master/examples/simple_demo.cpp

Oh, did not notice one need's preallocated arrays.
Thanks for the demo!

Glad it makes sense now. Thanks again for your interest. Let me know of any future questions. Happy to help!


_WD_IsLatestRelease() not works

after renaming this repository
This following:
https://github.com/Danp2/au3WebDriver/commit/28f0fd0a3032a68f618cedb138f24276a2bb34d4
should be immediately published as a new version.
I mean here:
https://github.com/Danp2/au3WebDriver/releases

But why? GitHub will do a redirection by default. No need to hurry or do I miss something?

I think you miss that InetRead() doesn't get redirected.

New release is out.


Subscription preview method

Related to:
https://github.com/crucialwebstudio/chargify-sdk-php/issues/36
API Reference:
https://reference.chargify.com/v1/subscriptions-preview
Tests passing ✅

Closing in favor of #37

Closing in favor of #37


Not compatible with python 3.11

Collecting pylbfgs
  Using cached PyLBFGS-<IP_ADDRESS>.tar.gz (98 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [56 lines of output]
      /tmp/pip-build-env-2g2k_p17/overlay/lib/python3.11/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-1_i61yy0/pylbfgs_7a13b0d8215f41f8952b32da42f79255/lbfgs/_lowlevel.pyx

Any updates on this?


Locally running Station web app cannot connect to locally running LocalTerra

Summary
Locally running Station web app (https://local.terra.money:3000/ ) cannot connect to locally running LocalTerra

Expected
It should work
Environments

Stage: ?
Chain: local testnet with LocalTerra
OS: macOS 10.15 Catalina
Platform: intel

Reproduce

Follow README in https://github.com/terra-money/LocalTerra , docker-compose up
Follow README in https://github.com/terra-money/station, npm run start
see error message window

@paulvi For me it is working well. Can you try to get some more information from chrome developer console?

Why is locally running app (on port :3000) trying to connect to :3060 ?
It should be some required port configuration, though I assume it should work by default when following README

Well app from https://localhost:3000/ can connect,
but not app from https://local.terra.money:3000/

Should anything from .env.development be applied? Or .env changed?
I tried all .env.development (including HOST=local.terra.money), it did not help.
Please point in right direction

do you have local.terra.money record in /etc/host ?

Yes, I have
(it is visible in screenshot above)

.env file does nothing

I ran successfully in a pretty hacky way.
under package.json
"start": "HOST=localhost react-scripts start"
we have to use https, without, the local storage was failing for me.

we could add a separate script for convenience purposes.
"start:local": "HOST=localhost react-scripts start"```

This way, if we wanted to test with local node, we can run `npm run start:local`

However, this is just the surface level fix.

A deep root level fix will involve setting an environment variable that captures the baseURL which the local terra node runs on.

Else, maybe we can open the local terra node to link with local.terra.money as well?

The latter sounds simpler.. Do we have infrastructure expertises with local terra node and if we are able to sym link local.terra.money instead of just localhost?

Actually the use of local.terra.money will be deprecated very soon. Just wait for a few days please.


Move ontology mappings that inform Frontend filters to S3 bucket

new, or leverage existing if we already have one that’s appropriate
Cell Type Ontology Mapping
Cell Type Descendants
Tissue Ontology Mapping
Tissue Descendants
Development Stage Ontology Mapping

@nayib-jose-gloria Can you also update the Tech Overview doc to make mention of this new bucket? Ty!


feat: Add markdown linter for curriculum, with checks of test YAML

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] All the files I changed are in the same world language (for example: only English changes, or only Chinese changes, etc.)
[x] My changes do not use shortened URLs or affiliate links.

This is a companion to #35735 and also addresses #35729.
@raisedadead suggested we should catch bad YAML syntax before  running the tests.  This is achieved by attempting to parse the YAML code blocks using the same library as challenge-md-parser and reporting any errors.
It also lints the markdown more generally, though with a lot of rules disabled.  This found a few problems that had not been reported (to my knowledge!), so it seems useful.  However, I'm not sure exactly which rules we want to be enforcing, and I don't know exactly where best to put these files, so any input would be great!

Hey @ojeytonwilliams can you rebase this PR for QA. Thanks for your patience.

@raisedadead No problem.  I'm not sure what (if anything) to do about the other languages, but it seems to be working fine for English.

This is a great start, I think we can throw this in a script (a new one or add into an existing one) in the location I have left in my comment in the review and take it from there. This way we are also able to use the LOCALE env var to get it working later with any language.

@raisedadead I feel daft now, because I must have skimmed your comments as I only just noticed the bit about tools/scripts/ci.  Sorry about that!  I'll take a look at that and see if I can merge the two and/or remove redundant stuff.
Also, this is supposed to fail the travis checks, because they now include the new script and there are a ton of challenges that have formatting issues.

Also, this is supposed to fail the travis checks, because they now include the new script and there are a ton of challenges that have formatting issues.

Yes, I understand, we might in that case fix up the formatting on this PR? Let's get others to work on this too?

Yes, I understand, we might in that case fix up the formatting on this PR? Let's get others to work on this too?

Yeah, we'd have to do it in this since we need newlines around the code fences.  I'd be more than happy if anyone wanted to help fixing the challenge files.

It seems that ensure-challenge-formatting.js is still useful in that it validates frontmatter.  However parsing the entire markdown seems redundant, given that exactly that happens during testing.  Regardless, this is feeling tangential to me, so maybe it would be better to just fix the linting in this PR
@RandellDawson @moT01 do you think you'll have time to look at fixing the challenge files?  If not, don't worry - just let me know and I'll figure it out.

We can use prettier for formatting and linting md-files.

I'm not sure we can.  As I understand it, prettier parses the code into an AST and then uses that to create new code.  The problem is that
text

```
code
```

generates a different AST from
text
```
code
```

so prettier will (I think) end up creating different output, rather than flagging up the second one as incorrect.

Do tests catch invalid yml-blocks?

They do.  However, I haven't written any tests to verify the linter is doing its job. I'll see what I can do.

I'm not sure we can. As I understand it, prettier parses the code into an AST and then uses that to create new code. The problem is that

I propose to add the next:
"scripts": {
    "format:other": "npm run prettier -- --write",
    "lint:other": "npm run prettier -- --check",
    "prettier": "prettier \"**/*.{md,css}\""
}
"lint-staged": {
    "*.{md,css}": [
      "prettier --write",
      "git add"
    ]
  }

also add .prettierignore file
node_modules
**/.cache
**/public
client/static

Do tests catch invalid yml-blocks?

They do. However, I haven't written any tests to verify the linter is doing its job. I'll see what I can do.

Do you mean current tests or lint checks from this PR?

Sure, I'll add the prettier stuff in the next commit.

Do tests catch invalid yml-blocks?

They do. However, I haven't written any tests to verify the linter is doing its job. I'll see what I can do.

Do you mean current tests or lint checks from this PR?

Both, actually.  Since https://github.com/freeCodeCamp/freeCodeCamp/pull/35735 the tests catch bad yml properly.  This PR just catches it (and other md issues) earlier.

I've added some tests and implemented @ValeraS's suggestions, with a couple of changes.  Firstly, node_modules is ignored by default, but it needed to ignore fixtures so it wouldn't 'fix' the tests!  Secondly, I went with --debug-check for format, since it seems like a good idea to err on the side of caution.
Anyway, adding prettier was a good call.  It cleans up some stuff I didn't expect it to and cuts down the amount of files we need to change manually.
For the moment, I haven't extended the markdown linter over the guide, since I don't know how to automatically fix files.

@ojeytonwilliams What lint errors do not fix a prettier?
We can check a yaml syntax simply by execute getChellangesFoLang.

prettier misses a bunch of stuff.  If you try running npm run lint:other and then npm run lint:challenges you can see some of them.
Some of the things prettier doesn't reliably catch are

Use of indentation instead of code fences
Lack of newline before a code fence
Multiple h1 elements
Skipping header numbers e.g. h2 -> h4

It's particularly bad if a newline doesn't appear before a code fence, as it will try to fix it. For example in curriculum/challenges/english/02-javascript-algorithms-and-data-structures/basic-data-structures/-iterate-through-the-keys-of-an-object-with-a-for...in-statement.english.md if you delete the empty line before the first code fence and npm run lint:other you end up with
...this could look like:
```js
for (let user in users) {
  console.log(user);
}

// logs:
Alan
Jeff
Sarah
Ryan

````

The with the closing four backticks wrapping the second fence.
I'm sure markdownlint will miss some mistakes, but it at least catches this.

One downside of prettier I've just discovered.  You can't write 'bad' code, it'll format it nicely regardless.  This is actually a problem, since at least one challenge demonstrates poorly formatted code to contrast with nicer code.
One PR I'm reviewing has the following challenge.  If we format it with prettier, it stops making sense.
You can see the before and after here
We can work around this by using <pre><code> blocks, see here, which prettier won't touch, but only if we know that's needed.
@raisedadead if it's just this challenge that does this, we're probably okay, but I don't know if that's the case.

I'm not 100% happy with the organisation of this, so any ideas are welcome.  The reasoning behind the two sets of linting rules is simply that the markdown in the guide is much looser than  in the curriculum.  If we lint new files in staging we can make sure new articles are nice and neat, but applying strict rules to the guide without some way of automating it is rather daunting.
The rules that it does enforce on the guide should be relatively easy to apply via regex.
I'll fix the conflict in a sec.

[...] This is actually a problem, since at least one challenge demonstrates poorly formatted code to contrast with nicer code [...]

&&

[...] If we format it with prettier, it stops making sense [...]

Can we have special escaping done somehow, like ESLint allows disabling rules per line, per file, etc.

[...] The reasoning behind the two sets of linting rules is simply that the markdown in the guide is much looser than in the curriculum. [...]

I can understand your frustration with this. But the real real solution to this would be to have a dedicated contributor editor (something we have been discussing and working on). That way we can cleanup the guide eventually. In all honesty the efforts spent on fixing formatting on Guides to be relaxed is fine, as long as we can improve the flow to make quick edits.
For now this is fine. Content editing is hard enough and doing it with Git is worse.

Can we have special escaping done somehow, like ESLint allows disabling rules per line, per file, etc.

We can and I should have thought of that!  Putting <!-- prettier-ignore --> before a code block will make prettier leave it alone.
However, the real problem is that we don't know which challenges need it.  Since we talked about this last I've realised that it's not just deliberately bad code that gets hit; it's anything that has slightly non-standard formatting.  Here are some examples
Before:

After:

Before:

After:

These aren't terrible, it's just they no-longer fully reflect the author's intent.  Also a lot of challenge seeds have extra empty lines for users to enter their code, which will get stripped out by prettier.
On balance, I don't think we should be using prettier on the guide or the challenges.

a dedicated contributor editor [...] That way we can cleanup the guide eventually. In all honesty the efforts spent on fixing formatting on Guides to be relaxed is fine [...]

An editor would be amazing!  Just to reiterate, I'm absolutely in favour of applying as strict linting as we can to future guide PRs.  That's why I added the 'full' linter to lint-staged, I just don't think it should be applied retroactively.

However, the real problem is that we don't know which challenges need it.

I think we can do an audit during the QA phase on beta.

On balance, I don't think we should be using prettier on the guide or the challenges.

I am inclined to that too. We want any linting and tooling to augment our work and not make it worse. I think your original approach is just as fine and better than anything.

I just don't think it should be applied retroactively.

I agree, and I think we should have some rules and not 'strict rules' as long as it's not platform code. It will be a way better experience than now.

Okay, cool.  I'll stop using prettier on markdown files (css seems safe) and get to work on the markdown itself so it can pass.  I think @RandellDawson's already sorted out the problems with prism languages, so we're well on the way.