Remove dependency conflicts in requirements.txt

Type of Issue
Select the type of issue:

[x] Bug report (to report a bug)
[ ] Feature request (to request an additional feature)
[ ] Tracker (I am just using this as a tracker)
[ ] Documentation Ask

Description
There are dependency conflicts in our requirements, and packages frozen on old versions.
This is affecting our env setup and will be a pain point for new users.
Current Behavior
Example of error from pip install:
(droidlet_test_env) [~/droidlet]$ pip install mypy-extensions==0.4.1             *[rebeccaqian_inference_script_fix]

Looking in indexes: https://pypi.org/simple, http://<IP_ADDRESS>
Collecting mypy-extensions==0.4.1
  Using cached mypy_extensions-0.4.1-py2.py3-none-any.whl (3.6 kB)
Installing collected packages: mypy-extensions
  Attempting uninstall: mypy-extensions
    Found existing installation: mypy-extensions 0.4.3
    Uninstalling mypy-extensions-0.4.3:
      Successfully uninstalled mypy-extensions-0.4.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mypy 0.800 requires mypy-extensions<0.5.0,>=0.4.3, but you have mypy-extensions 0.4.1 which is incompatible.
Successfully installed mypy-extensions-0.4.1

Expected Behavior
A new user should be able to create a new env and install requirements relatively smoothly. This is something that we will need to maintain regularly.
Steps to reproduce
conda create --name [NEW_ENV]
conda activate [NEW_ENV]
pip install -r requirements.txt

Thanks for creating the issue. Which anaconda module are you using ?

Thanks for creating the issue. Which anaconda module are you using ?

@snyxan can you take this one ?


Two bogus specs for ServiceTemplateAnsiblePlaybook

If you enable partial double verification in the rspec configuration, it will reveal these two failures, which I've partially snipped for brevity:
 1) ServiceTemplateAnsiblePlaybook#update_catalog_item uses the existing dialog if :dialog_id is passed in
     Failure/Error: expect(described_class).to receive(:create_new_dialog).never
       ServiceTemplateAnsiblePlaybook does not implement: create_new_dialog
     # ./spec/models/service_template_ansible_playbook_spec.rb:243:in `block (3 levels) in <top (required)>'

  2) ServiceTemplateAnsiblePlaybook#delete_job_templates does not destroy a job template if there is no associated configuration_template
     Failure/Error:
       expect(ManageIQ::Providers::EmbeddedAnsible::AutomationManager::ConfigurationScript)
         .to receive(:delete_in_provider_queue).never

       ManageIQ::Providers::EmbeddedAnsible::AutomationManager::ConfigurationScript does not implement: delete_in_provider_queue
     # ./spec/models/service_template_ansible_playbook_spec.rb:294:in `block (3 levels) in <top (required)>'

The issue is that you're testing against singleton methods that don't exist. Those are instance methods.
This raises the question on whether or not create_new_dialog and delete_in_provider_queue are supposed to be a singleton methods, or if the specs are just testing against the wrong thing.

@syncrou git blame says this is you. :)


Cooking game

Description
allows players to follow detailed recipes to cook virtual dishes. This feature will enhance gameplay by providing a more immersive and educational cooking experience.Implement a system where players can select and follow recipes step-by-step to create virtual dishes.
Enhance the educational value of the game by teaching players real cooking techniques and recipes.
Recipe Selection
Add a new menu option where players can browse and select from a variety of recipes.
Step-by-Step Instructions
Provide clear, detailed instructions for each step in the recipe.
Ingredients and Tools
List required ingredients and kitchen tools for each recipe.
Ensure the interface is accessible and easy to navigate, especially during the cooking process.
Fixes:  #1808
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist:

[X] I have made this from my own
[X] I have taken help from some online resourses
[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[] I have made corresponding changes to the documentation
[X] My changes generate no new warnings

ATTACH SCREEN-SHOTS / DEPLOYMENT LINK

Hey @mahek0620 @Sulagna-Dutta-Roy
I am unable to pull origin or clone the forked repository because there is an invalid path in your file. Could you please resolve this issue?

@mehak @Sulagna-Dutta-Roy pls resolve this issue


Show output also for failed tasks?

Using shell2http 1.4 (a fantastic tool - thanks a lot for creating this!) we are observing the following behaviour: that the response for tasks that return a non-zero exit code is simply something like:
exec error: exit status 2

In order to actually see the error message and yet still know that the task has actually failed, we are resorting to something along the lines of:
/my/task '(do_stuff; echo Exit code: $?; exit 0)'

Would this be a good candidate for a command-line flag? Or is there an easier way to achieve the desired result (i.e. having stdout and/or stderr returned even for non-zero exit codes) without the hackery in the example above?

Yes, it makes sense to add the option on the command line like -output-errors (or -out-on-fail...).
And may be another option -output-stderr, for return stderr-out of script (now stderr is printed to the console).

And may be another option -output-stderr, for return stderr-out of script (now stderr is printed to the console).

Ah, good idea! So two orthogonal options?
-show-errors: returns the standard output even if the command exits with a non-zero exit code
-include-stderr: also returns output written to stderr (default is stdout only)

Or would -show-errors automatically imply -include-stderr? I assume the users would want both most of the time since the error message would likely go to stderr. But I guess there are also enough cases where that is not the case that one might want to make the options separate?
I can also imagine cases where you just want -include-stderr, such as for commands that print diagnostic info to stderr even when they do not fail

Yes, two orthogonal options.
Thanks for naming for options:)

Implemented in #4, #5


Improve build rules for (test only) library quic_port_utils.

Description:
Improve build rules for (test only) library quic_port_utils, by removing the intermediate library //extensions/quic_listeners/quiche/platform:quic_platform_port_utils_impl_lib.
Risk Level: none, build only
Testing:
bazel test --test_output=all test/extensions/quic_listeners/quiche/platform:all @com_googlesource_quiche//:all
bazel test --test_output=all --define quiche=enabled test/extensions/quic_listeners/quiche/platform:all @com_googlesource_quiche//:all
Docs Changes: none
Release Notes: none

/assign @danzh2010

/retest

/assign @htuch

/assign @alyssawilk


Actions to Azure via OIDC - unnecessarily secret?

Is there a reason why we instruct folks to mark these variables as secrets?  It seems unnecessary and the point of the whole flow is that "you don't need secrets anymore!" 😁
            client-id: ${{ secrets.AZURE_CLIENT_ID }}
            tenant-id: ${{ secrets.AZURE_TENANT_ID }}
            subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 081be8c9-7871-db69-977c-9304daa6c870
Version Independent ID: aa3e5074-a630-3756-be86-295d1a2d101f
Content: Connect GitHub and Azure
Content Source: articles/github/connect-from-azure.md
Service: azure
GitHub Login: @N-Usha
Microsoft Alias: ushan

Thanks, closing for now because we shared this feedback with the product team.
#please-close


Calico.yaml file is not applying correctly: error

Hi, I am configureing Multi-node Kubernetes Cluster on ubuntu in VirtualBox . While apply calico.yaml file. I got error while creating last pods. Error is attached.
I am using this file:
curl https://docs.projectcalico.org/manifests/calico.yaml -O

uncomment " CALICO_IPV4POOL_CIDR" and its value as well (<IP_ADDRESS>/16)
while applying calico got error:
error:
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers unchanged
serviceaccount/calico-kube-controllers unchanged
error: resource mapping not found for name: "calico-kube-controllers" namespace: "kube-system" from "calico.yaml": no matches for kind "PodDisruptionBudget" in version "policy/v1"
ensure CRDs are installed first

I already checked calico-kube-controllers and PodDisruptionBudget"  it exist in yaml file.

@iqrazaf what version of Kubenretes are you using? The PodDisruptionBudget v1 API was introduced in k8s v1.21 so requires that release at a minimum.

Yep. You need to use v1.21 Kubernetes or later.

I have the same trouble. Finally, i find it is the appVersion info mistake in YAML file. I write the ’V1‘, which should be 'v1' in my project. Hope it is useful for you.

Thank everyone. Problem solved. It was a version differences issue for Docker and Kubernetes

@iqrazaf Can you please tell what solved the problem.

@sshveta: Can you please tell what solved the problem.
It was a version differences issue for Docker and Kubernetes

Can u tell exactly, i also got the same issue.

Can u tell exactly, i also got the same issue.

install latest version of Docker and Kubernetes and hold the Kubernetes version after installation.
Install Docker (All hosts)
sudo apt-get install docker.io -y
Update the package repository and Install K8S components (All hosts):
sudo apt-get update
sudo apt-get install -y kubelet kubectl kubeadm
sudo apt-mark hold kubelet kubeadm kubectl


Is it possible to have nodes with outlines?

Is it possible to have nodes with outlines in bubbleboxer? I'm hoping to create something similar to the soft-serve "git selector" + "git viewport" UI.

So what you can do is to embed a Boxer instance into a Model which draws a outline around it.
When you use this Model in a Leaf of a other Boxer instance, you have a outline around the contained nodes and leafs.
But at the moment its not directly possible.


Time forfeits

Mister Queen exceeds time in about 80 % of its games around move 40. Tried with 2+1, 40/2, with and without internal book, no difference.
BTW, there is a fluctuation in memory allocation between 26 and 67 MB's. It uses the bigger memory when on move.

Just now after all the time I saw your issue Gabor and looked at MisterQueens time management.
Actually it has none, but a hard coded limit of exactly 4 seconds per move :)
void handle_go(char *line) {
    search.uci = 1;
    search.use_book = 1;
    search.duration = 4;
    char *key;
    char *token = tokenize(line, " ", &key);
    while (token) {
        if (strcmp(token, "infinite") == 0) {
            search.duration = 0;
            search.use_book = 0;
        }
        else if (strcmp(token, "movetime") == 0) {
            char *arg = tokenize(NULL, " ", &key);
            search.duration = atoi(arg) / 1000.0;
        }
        else if (strcmp(token, "ponder") == 0) {
            return; // no pondering yet
        }
        token = tokenize(NULL, " ", &key);
    }
    thread_start();
}

That's all, it does not know about btime/wtime/inc or book/GUI moves
You could compile diff. versions with other limits w/o having to add some own timecontrol code, though.

Thanks Günther. It's funny I did not notice that, it's the next line below search.use_book = 1 which I changed to 0 to disable the book (see the relevant issue).


brain FNO example

Modulus Pull Request
Description
Brain example 1st PR
Checklist

[x] I am familiar with the Contributing Guidelines.
[x] New or existing tests cover these changes.
[x] The documentation is up to date with these changes.
[x] The CHANGELOG.md is
up to date with these changes.

Dependencies

Closing since merged in PR: https://github.com/NVIDIA/modulus/pull/242


LUIS Integration

How can I iplement the LUIS service to the FAQ bot? Is it possible?

Querying from QnAmaker is referred here: https://github.com/OfficeDev/microsoft-teams-apps-faqplus/blob/675c435fd0ea1695d37717a69f507f96602490f3/Source/Microsoft.Teams.Apps.FAQPlusPlus/Bots/FaqPlusPlusBot.cs#L787
And you may need to update the code to implement LUIS


Warning: Cannot update a component (ForwardRef(ReactQueryDevtoolsPanel)) while rendering a different component

When using queryCache.setQueryData in a component that is using useQuery, React throw this warning
Warning: Cannot update a component (`ForwardRef(ReactQueryDevtoolsPanel)`) while rendering a different component (`Todos`). To locate the bad setState() call inside `Todos`, follow the stack trace as described in https://fb.me/setstate-in-render
    in Todos (at src/index.js:144)
    in div (at src/index.js:143)
    in div (at src/index.js:134)
    in App (at src/index.js:121)
    in ReactQueryConfigProvider (at src/index.js:57)
    in Root (at src/index.js:432)

Example :
https://codesandbox.io/s/epic-diffie-swiyq?file=/src/index.js:4415-4466

This is now fixed with the latest versions of both react-query and react-query-devtools

Thanks you ! 🎉

I'm still facing to this issue:
"react-query": "2.12.1",
"react-query-devtools": "2.4.4",

simplify code:
<ReactQueryConfigProvider config={{}}>
	<Suspense fallback={<LoadingLayout />}>
		<ThroughProvider>
			<AuthContext.Provider value={authContext}>
				<Homepage>
					...
					<Test />
					...
				</Homepage>
			</AuthContext.Provider>
		</ThroughProvider>
		{process.env.NODE_ENV === 'development' && <ReactQueryDevtools initialIsOpen />}
	</Suspense>
</ReactQueryConfigProvider>

error message:
Warning: Cannot update a component (`ForwardRef(ReactQueryDevtoolsPanel)`) while rendering a different component (`Test`). To locate the bad setState() call inside `Test`, follow the stack trace as described in https://fb.me/setstate-in-render
    in Test (at Homepage.tsx:94)
    in div (at DefaultContainer.tsx:8)
    in DefaultContainer (at Homepage.tsx:84)
    in Homepage (at App.tsx:104)
    in ThroughProvider (at App.tsx:102)
    in App (at src/index.tsx:25)
    in Suspense (at src/index.tsx:24)
    in ReactQueryConfigProvider (at src/index.tsx:23)

  | console.<computed> | @ | index.js:1
  | overrideMethod | @ | react_devtools_backend.js:2273
  | printWarning | @ | react-dom.development.js:88
  | error | @ | react-dom.development.js:60
  | warnAboutRenderPhaseUpdatesInDEV | @ | react-dom.development.js:23241
  | scheduleUpdateOnFiber | @ | react-dom.development.js:21165
  | dispatchAction | @ | react-dom.development.js:15660
  | (anonymous) | @ | index.js:322
  | (anonymous) | @ | queryCache.ts:93
  | notifyGlobalListeners | @ | queryCache.ts:93
  | buildQuery | @ | queryCache.ts:236
  | updateQuery | @ | queryObserver.ts:208
  | QueryObserver | @ | queryObserver.ts:34
  | useBaseQuery | @ | useBaseQuery.ts:19
  | usePaginatedQuery | @ | usePaginatedQuery.ts:60
  | useApiPaginatedQuery | @ | licenses.tsx:26
  | useApiLicenses | @ | licenses.tsx:40
  | Test | @ | Homepage.tsx:23

please, do you have any idea, what I do wrong?


Error: Can't find Python executable

while im trying to install node_modules i get this error:
Error: Can't find Python executable "C:\Users\Tal\AppData\Local\Programs\Python", you can set the PYTHON env variable.
it looks like you have a PYTHON enviroment variable on your local machine and i cant run the program without it

@TalArbatov do you have python installed ?

Oh right, not on my new machine...
It could be a good idea to mention Python in the README or the installation guide inside it


[stable/moodle] ingress,existingClaims,affinity,resourcesIncreased,r4ImageVerUpdate

Tested and:

added:

ingress (http & tls)
affinity
support for existingClaim (persistent volume claim)

updated:

resources Increased
image bump from r3 to r4

Hi @sameersbn,
Can you take a look at this ? Let me know if you have any questions.

/ok-to-test

Pleas bump the minor version of the chart.

fixes have been done per requested changes to both readme and bumped the chart versions.

Can you please move the updates to stable/mariadb chart in its own pull-request please?

Hi, I've updated the pull-request. Please take a look at https://github.com/ReSearchITEng/charts/pull/1, which creates a single mount at /bitnami for the persistence of the moodle application. It would be nice to have in included in this pull-request.
Lastly, please move the stable/mariadb chart changes to a separate pull-request.

due to typo in author of the commit, CLA does not validate. I am forced to recreate the repo and create a different PR.

this is the new PR: https://github.com/kubernetes/charts/pull/1534


windows support

not something on the roadmap, unfortunately! i'll be keeping LlamaChat native to Apple's platforms


Voice Command

I want to make a ordering skill and for that I need to take repeatedly input many times , how can I do that?

Take a look at the 'remember' and 'configure' skills. They do that pretty well.


Getting issue in decrypting with own private key

Hi @jchavannes ,
thanks for creating such a useful package,
while trying encrypt_test
I just replaced the content with my public and private gpg keys but getting error while decoding in my case as below
Error reading message: openpgp: incorrect key   and decrypted message is empty , I checked the encrypted data ,looks fine.
help appriciated .
[Note] have exported gpg keys from same linux machine with below commands
Public Key - gpg --armor --export <keyid>
Private Key - gpg --armor --export-secret-keys <keyid>

Hi @Naveenkhasyap I'm not sure why you might be having that issue. Unfortunately I have not used this repo in awhile and don't have more info that could help.

No issues, thanks for the prompt reply


Make sorting more intuitive

The arrows are aligned in a strange way, making it confusing how to sort columns.
OLD

NEW/FIXED

@cuahsimarko, we should not be making changes to 3rd party resource files. This rule should rather be overwritten in one of our CSS files. Also, could you point out which issue this is referencing?

There is no issue, I ran into this problem when working with CZO views.

@Maurier I'm aware of that, but since we don't have a frontend grunt webpack or other build process to generate minified files, interact with vendor js scripts, I was just trying to get a usability fix out for users, since the current UX is pretty bad...

The file seemed like it was modified or corrupted previously, repeating some elements. Let's chat offline about updating jQuery anyway, getting a fresh vendor file and then implementing a proper CSS override, as you've suggested.

Discussed different path forward with Mauriel will take 3-4 sprints to update jQuery then revisit... though it's likely handled already in the vendor updates


TypeError: Cannot read property 'toLowerCase' of undefined

Version
2.5.16
Reproduction link
I can't create a minimal reproduction
Steps to reproduce
I don't know how to reproduce that, it appears sometimes when I reload a page, and I can't create a minimal reproduction...
What is expected?
Expects the page reload normally.
What is actually happening?
Sometimes while page is reloading, this error appears in the browser:

And this error appears in the console:
at Object.<anonymous> (.nuxt/server.js:1:0)
at __webpack_require__ (webpack:/webpack/bootstrap 9148d4a2990df53d293e:25:0)
at server-bundle.js:92:18
at Object.<anonymous> (server-bundle.js:95:10)
at evaluateModule (/PROJECT_FOLDER/node_modules/vue-server-renderer/build.js:8338:21)
at /PROJECT_FOLDER/node_modules/vue-server-renderer/build.js:8396:18
at new Promise (<anonymous>)
at /PROJECT_FOLDER/node_modules/vue-server-renderer/build.js:8388:14
at Object.renderToString (/PROJECT_FOLDER/node_modules/vue-server-renderer/build.js:8564:9)
at Renderer.renderRoute (/PROJECT_FOLDER/node_modules/nuxt/lib/core/renderer.js:344:41)
at Renderer.nuxtMiddleware (/PROJECT_FOLDER/node_modules/nuxt/lib/core/middleware/nuxt.js:12:31)
at call (/PROJECT_FOLDER/node_modules/connect/index.js:239:7)
at next (/PROJECT_FOLDER/node_modules/connect/index.js:183:5)
at SendStream.error (/PROJECT_FOLDER/node_modules/serve-static/index.js:121:7)
at emitOne (events.js:116:13)
at SendStream.emit (events.js:211:7)
at SendStream.error (/PROJECT_FOLDER/node_modules/send/index.js:270:17)
at SendStream.onStatError (/PROJECT_FOLDER/node_modules/send/index.js:421:12)
at next (/PROJECT_FOLDER/node_modules/send/index.js:736:16)
at onstat (/PROJECT_FOLDER/node_modules/send/index.js:725:14) statusCode: 500, name: 'TypeError' }

This error appeared a week ago and since then I can't found a cause for it.

We need a minimal reproduction. you can try using the forum, the Discord server or StackOverflow for questions first. But feel free to come back and open an issue if it turns out to be a bug 🙂


JPG resolution drops to 96 dpi

Hi,
I have a few scripts that I'm using to flip and rotate JPG files.
Below is the rotating function:
async function rotateImage(targetFilePath, deg = 0) {
    const sharpImage = sharp(targetFilePath)
    sharpImage.rotate(deg);
    const buffer = await sharpImage.toBuffer();
    await sharp(buffer).toFile(targetFilePath)
}

The image I'm processing is available here:
https://drive.google.com/file/d/1WmqFj14LaUA0OAgWiiTgWUiTfLDaCqwj/view?usp=sharing
The rotation works well, but it seems that the images lose resolution and drop from 600 dpi to 96 dpi.
Is there a way I can keep the original image's resolution while processing it?
Thanks!

JPEG doesn't have a concept of resolution, but this image has EXIF metadata:
$ vipsheader -a I600.jpg | grep Resolution
exif-ifd0-XResolution: 600000/1000 (600.000, Rational, 1 components, 8 bytes)
exif-ifd0-YResolution: 600000/1000 (600.000, Rational, 1 components, 8 bytes)
exif-ifd0-ResolutionUnit: 2 (Inch, Short, 1 components, 2 bytes)

Please see https://sharp.pixelplumbing.com/api-output#withmetadata for control over this.

I've tried adding the withMetadata() method, to keep all original exif like so, but it doesn't seem to work.
async function rotateImage(targetFilePath, deg = 0) {
    const sharpImage = sharp(targetFilePath).withMetadata();

    sharpImage.rotate(deg);
    const buffer = await sharpImage.toBuffer();
    await sharp(buffer).toFile(targetFilePath)
}

I've also tried dictating a hardcoded value for the X and Y resolution, as attached below, but the result is still  96dpi:
async function rotateImage(targetFilePath, deg = 0) {
    const exifObject = {
        exif: {
            IFD0: {
                image: {
                    XResolution: 600,
                    YResolution: 600
                }
            }
        }
    };
    const sharpImage = sharp(targetFilePath).withMetadata(exifObject);
    console.dir(sharpImage);
    sharpImage.rotate(deg);
    const buffer = await sharpImage.toBuffer();
    await sharp(buffer).toFile(targetFilePath)
}

Is there a working example for retaining both of these values?

Did you see the example at https://sharp.pixelplumbing.com/api-output#withmetadata
// Set output metadata to 96 DPI
const data = await sharp(input)
  .withMetadata({ density: 96 })
  .toBuffer();

(I need to fix the formatting of this example in the docs as it's currently being displayed as part of the previous example.)

I did, neither the density property did not seem to have any affect over the output file's x and y resolution.
I also looked in the output unit tests and tries to imitate the way the exif should have been dictated, but it didn't seem to work either.
Is this the proper way to describe the exif data?
{
        exif: {
            IFD0: {

                XResolution: '600dpi',
                YResolution: '600dpi'
            }
        }

I did, neither the density property did not seem to have any affect over the output file's x and y resolution.
I also looked in the output unit tests and tries to imitate the way the exif should have been dictated, but it didn't seem to work either.
Is this the proper way to describe the exif data?
{
        exif: {
            IFD0: {

                XResolution: '600dpi',
                YResolution: '600dpi'
            }
        }

@AdiLew Were you able to make any progress with this? Did you try .withMetadata({ density: 600 })?

Closing due to inactivity but please feel free to reopen with more details if further help is required.


Add Advantech interfaces

Add new interfaces below:

EPC-T4286
EPC-P3000
AIIS-3410P
SOM-6882
IPC-220
UTC-520

@paul53huang Can you please update your PR branch (sync with origin/main), or allow maintainers to edit the PR so we can update it for you? Thank you.

Hi @rido-min ,
For now, we would like to use the same interface config for these devices.
The reason is that, It's better for us to schedule update each device.
Thanks for checking this commitment.
We'll keep updating devices interfaces.
Best Regards,
Paul


Charging Profiles - convertFromVendorChargingSchedule does not work for profiles pushed in watts

Limit for the composite schedule is returned in amps from the station.
The value is divided by the voltage, because the server thinks it is returned in watts.
(appearing on dc station)

SAP-Mougins-15 uses Watt in the template and we push the charging profile in Watts and the getCompositeSchedule() returns Amps !?

We must align the charger config via OCPP params to use the right unit provided in the Template in both setting and reading the charging profiles.
@jerome-benoit Could you check?

If you are sure that the unit specified in the profile is properly set to Watt, then it's a firmware bug.
There's no OCCP parameter to set, everything is in the charging profile. The read-only parameter ChargingScheduleAllowedChargingRateUnit have Current,Power value.
The current firmware version should support correctly A as a unit, so the workaround is to switch to A in the template.

If you are sure that the unit specified in the profile is properly set to Watt, then it's a firmware bug.
There's no OCCP parameter to set, everything is in the charging profile. The read-only parameter ChargingScheduleAllowedChargingRateUnit have Current,Power value.
The current firmware version should support correctly A as a unit, so the workaround is to switch to A in the template.

Yes I'm sure it is set in Watt because this charger supported only charging profile in Watt at the time we set it up and we never changed that.
With new firmware updates according this OCPP param, they now support Amps:

And I think we should rely on the OCPP param to check which unit to use, and then we can remove it from the template:

When we request the profile, do we use the param chargingRateUnit in the getCompositeSchedule?

Let me know!
But there is room for improvement here.

The bug is on our side:
[
      {
            "chargingProfile": {
                  "chargingStationID": "SAP-Mougins-15",
                  "connectorID": 2,
                  "chargePointID": 1,
                  "profile": {
                        "chargingProfileId": 2,
                        "chargingProfileKind": "Absolute",
                        "chargingProfilePurpose": "TxProfile",
                        "transactionId":<PHONE_NUMBER>,
                        "stackLevel": 2,
                        "chargingSchedule": {
                              "chargingRateUnit": "A",
                              "chargingSchedulePeriod": [
                                    {
                                          "startPeriod": 0,
                                          "limit": 566
                                    },
                                    {
                                          "startPeriod": 900,
                                          "limit": 448
                                    },
                                    {
                                          "startPeriod": 1800,
                                          "limit": 0
                                    }
                              ],
                              "startSchedule": "2020-10-16T14:30:00.000Z",
                              "duration": 2700
                        }
                  }
            }
      }
]

The charge unit and value should be in Watt as defined in the template.

Okay, this is a regression.
@OliveGerste Could you have a look?

@jerome-benoit @LucasBrazi06
We push the cp in watts but it is always saved and logged in amps, because the conversion is done in the vendor implementation and the saving is done in OCPPUtils.
So not really a bug, but a bit confusing maybe.

@jerome-benoit @LucasBrazi06
We push the cp in watts but it is always saved and logged in amps, because the conversion is done in the vendor implementation and the saving is done in OCPPUtils.
So not really a bug, but a bit confusing maybe.

Exact! Forgot that point indeed!
@jerome-benoit You'll see the Watts in the real OCPP charging plan pushed to the charging station.
@OliveGerste Do we use the chargingRateUnit when we request the composite schedule?

@LucasBrazi06 Yes we use the chargingRateUnit. That is why the values from the station are so weird.
limitWatts: 651 --> this is the limit returned from the station
limit: 2.8 --> this is what we convert with the chargingRateUnit (division by 230V) (method: convertFromVendorChargingSchedule)

Yes we use the chargingRateUnit. That is why the values from the station are so weird.
limitWatts: 651 --> this is the limit returned from the station
limit: 2.8 --> this is what we convert with the chargingRateUnit (division by 230V) (method: convertFromVendorChargingSchedule)

Then if we request the Charging Profile using Watt in getCompositeSchedule and we got Amps instead of Watts, then this is a bug :-), agree?

In my understanding yes. This is what is returned by the cs unfiltered:

In my understanding yes. This is what is returned by the cs unfiltered:

Ok thanks!
@jerome-benoit Could you contact Delta and open them an issue?
Thanks!

I take over that issue then but in asana.
For the automation that can be done from OCPP parameter, let's discuss it later.

Latest firmware version 3.3.6 installed on Delta charging station.
@OliveGerste could you please check if the issue is fixed or not?

@jerome-benoit
Seems not to be fixed. It is still returned in amps per phase:


Akka HTTP

Repo in progress: https://github.com/1984weed/akka-http-realworld-example

@ 1984weed lets use this issue to track progress. Also, sent out a logo for Akka HTTP impl to your repo. Good luck!

I left only some refactorings

@anishkny
I just completed the functionality.
https://github.com/1984weed/akka-http-realworld-example
How should I do next?

Looks great! Could you fix failing build:

Oh. Sorry. I made a mistake.
I fixed it. @anishkny
Thanks 🙇

How do I have to do to go to the next step?

@1984weed to ensure API correctness, could you run tests defined here against your backend in CI: https://github.com/gothinkster/realworld/tree/master/api

Thanks. I just added the API test to CI !!
https://github.com/1984weed/akka-http-realworld-example/blob/9824fc701d1476eae2b389d2410881af1577b147/.travis.yml
Is it OK?

Great work. LGTM PTAL @Cameron-C-Chapman


wazero: updates with better close function

Most recent code exits the module with a sys.ExitError, if something
compiled WASI in a way that invokes "proc_exit". While your code isn't
doing that, it is helpful to have the update in case a future change
does :D

Thx :)


Use large_image to subsample imagery

Switch the subsampling endpoints/backend to leverage large_image rather than the code in rgd/geodata/models/imagery/subsample.py

large_image has a getRegion function.  It can take the corner points (in pixel space or in a projection's coordinates) and the desired output resolution or magnification.  It outputs a rectangular image or NumPy array.
Since getRegion doesn't handle GeoJSON in any way -- just min/max values in x and y in pixel or projection coordinates, we will simply use the min/max bounds of the GeoJSON geometry for these endpoints.
Work to be done in RGD

[ ] Create new endpoints that accept an image ID and four values for min/max coords

This will need to have two versions: image vs world coords

api/geodata/imagery/image_entry/<int:pk>/region/world/<xmin>/<xmax>/<ymin>/<ymax>
api/geodata/imagery/image_entry/<int:pk>/region/pixel/<umin>/<umax>/<vmin>/<vmax>

Use large_image's getRegion to handle generating a sumbsampled image from the given parameters
Return the extracted region image file from the endpoint

can this be done in the request thread or should this be a task? cc @manthey
If a task, we may need to create a new ImageFile and ImageEntry. Note that this issue is why I added SubsampledImage in the first place

[ ] Create another endpoint for using an annotation as the ROI

api/geodata/imagery/image_entry/<int:pk>/region/annotation/<int:annotation_id>
This will basically wrap the pixel coordinates ROI endpoint with the min/max of the annotation bounds

[ ] Remove SubsampledImage model (if necessary), rgd/geodata/models/imagery/subsample.py module, and associated tasks

Work to be done directly in large_image

[ ] The output of getRegion() doesn't have any geospatial information attached to it -- we need to set it to output the region as a Cloud Optimized GeoTiff (or regular GeoTiff)

Based on the technical requirements, I may be misunderstanding the purpose of this feature

I thought the point of this was to create subsamples for later retrieval.  If creating ImageFile and ImageEntry is optional, then the caller must expect immediate results.  Should this endpoint be idempotent and return a previously generated sub-sample if exists?
Likewise, I don't see how removing SubsampledImage is an option.  You need somewhere to store the pointer to the ImageEntry with the parameters that created it.  I think you can remove all the extra stuff that goes along with that model and have it be a normal rest model that triggers a job.
If this happens in a worker job, there needs to be a new endpoint not listed here.  The two you listed generate new SubsampledImage models that trigger jobs to be created.  They return the model, which has a nullable URI reference to an ImageEntry which will be null.  Hitting the same endpoint again (or maybe adding a get-by-id endpoint for subsampled imagery) will have to be tried until the job completes and the new URI populates.  Is this correct?
Won't creating a new ImageEntry cause duplicate data to appear in the search results?  Does a new field need to be added to indicate that an image is not a derived product?  Should a left outer join be performed?

Answers:

No need to remove Subsampled Image.
No need to create new imagefile or imageentry yet.  Data can go directly into a checksum file.
Mostly this work just involves replacing the task_funcs to use large_image and remove most of what's in subsample.py

(for future) I am wondering if @manthey will consider adding support for taking geojson as input for getRegion?

Currently large_image getRegion can take coordinates in a variety of units (pixel space, physical distance in pixel space, any projection for geospatial entries).  It can optionally scale the results.  It currently only outputs a numpy array or an image and has the limitation that the entire region has to fit in memory (and, if outputting an image, the image has to be able to be created with PIL, which effectively limits it to 1 gigapixel).  Some of the output can be transparent (of, more precisely, the output is likely to be in RGBA or LA color space).
To properly support RGD, we need to output a COG (not a tif through PIL) which includes geolocation data.  For generality, for non-geospatial data large_image's getRegion should also be able to output pyramidal tiffs.  Conceptually this isn't hard.
getRegion currently outputs a single image, I think with the limitation that the output is never more than 4 channels (RGBA), but I'll have to confirm that is true with numpy output as well.  For hyperspectral files, this means that we might be picking no more than three channels (though you can composite the data as part of the process via the style options).  For palettized files (e.g., land use data where the pixel values are categories), getRegion outputs still outputs an RGBA image -- this means that the categorical information can be lost.  If we need to preserve all of this, it means that when we output a region from large_image, the gdal tilesource might override the getRegion method when COG is requested and internally use GDAL to do the work.  I think we'd only ever want to do this if no styling was applied (i.e., the bands aren't being remapped as part of the getRegion request).
If we add geoJSON region selection, I'd want to support that for all tile sources.

See https://github.com/girder/large_image/issues/567

See https://github.com/girder/large_image/pull/594.

@banesullivan The PR on large_image is ready to be used and has basic tests.  You can pull from that PRs branch to try things out.

Great! Thanks for the update, I will start testing this


Some selections in Admin modal need a brief description of the function

These three need a small description of what they do that appears when you select them from the drop down menu in the Admin modal.

Drain Agreement
Rescue Tokens
Rescue Eth

The other's are self explanatory.

Fixed in PR #41


Fix tensorflow min version depending on Python version

Fix tensorflow min version depending on Python version.
Related to:

#6991

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Unexpected behaviour of runt cost with long words

Description
When setting the runt cost to an extremely high value, I would expect there to always be at least two words in the last line. However, it seems that when the last word is so long that it contains a hyphenation opportunity, it doesn't "glue together" the last two words, but instead the last two parts of the hyphenated word. For example:
#set page(width: 100pt, height: auto)
#set text(costs: (runt:<PHONE_NUMBER>000000000%))
#set par(justify: true)

// Works as expected, as "bad" is short
is it good or is it bad

// Doesn't work as expected, "issue" is alone on its line, as it can be hyphenated as "is-sue"
is it good or is it issue

// Works as expected, as boxes can't be hyphenated
is it good or is it #box[issue]

// Doesn't work as expected, "malicious" is even hyphenated
is it good or malicious

(Also tested after 711a2d88079f0b3929bbb6be97bda52139e22aa1)
Reproduction URL
No response
Operating system
Web app
Typst version

[X] I am using the latest version of Typst

While the runt cost is high, the cost of the resulting awful word spacing on the first line may be bad enough to offset that.
(I'm only speculating.)

While the runt cost is high, the cost of the resulting extremely awful word spacing on the first line may be bad enough to offset that.

When using an equally long monosyllabic word (e.g. "scratched" instead of "malicious"), the awful word spacing is accepted, so I think it has something to do with hyphenation.


Client Allocation Request for: younchang

Client Allocation Questions
Core Information

Name: younchang
Website / Social Media: http://younchang.co.kr/en/
Region: North America
DataCap Requested: 10TiB
Addresses to be Notarized: f1igeqr2jbvdjozbw4yozdnixrzwpdamduv6negia
Notary Requested: Broz221

@trunbyevi Please subscribe to notifications for this Issue to be aware of updates. Notaries may request additional information on the Issue.

Thanks for your application!
Could you please introduce you and your company?

Hi, I viewed your website and found out you are from a Korea company. So I suggest you to choose an Asia notary.


Please clarify what version of Node.js is needed

I happened to have v18.16.0 installed due to other things i'm working on, and on a fresh clone, I get:
dowideit.sven@ABC-KC00PW16GC k6 % npm install
npm WARN deprecated<EMAIL_ADDRESS>Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)

added 340 packages, and audited 341 packages in 6s

25 packages are looking for funding
  run `npm fund` for details

7 vulnerabilities (2 moderate, 4 high, 1 critical)

To address issues that do not require attention, run:
  npm audit fix

To address all issues, run:
  npm audit fix --force

Run `npm audit` for details.
dowideit.sven@ABC-KC00PW16GC k6 % npm start

><EMAIL_ADDRESS>start
> webpack

Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db

Why you should do it regularly:
https://github.com/browserslist/browserslist#browsers-data-updating
node:internal/crypto/hash:71
  this[kHandle] = new _Hash(algorithm, xofLen);
                  ^

Error: error:0308010C:digital envelope routines::unsupported
    at new Hash (node:internal/crypto/hash:71:19)
    at Object.createHash (node:crypto:133:10)
    at BulkUpdateDecorator.hashFactory (/Users/dowideit.sven/src/rockset/k6/node_modules/webpack/lib/util/createHash.js:144:18)
    at BulkUpdateDecorator.update (/Users/dowideit.sven/src/rockset/k6/node_modules/webpack/lib/util/createHash.js:46:50)
    at SourceMapSource.updateHash (/Users/dowideit.sven/src/rockset/k6/node_modules/webpack-sources/lib/SourceMapSource.js:233:8)
    at NormalModule._initBuildHash (/Users/dowideit.sven/src/rockset/k6/node_modules/webpack/lib/NormalModule.js:839:17)
    at handleParseResult (/Users/dowideit.sven/src/rockset/k6/node_modules/webpack/lib/NormalModule.js:904:10)
    at /Users/dowideit.sven/src/rockset/k6/node_modules/webpack/lib/NormalModule.js:995:4
    at processResult (/Users/dowideit.sven/src/rockset/k6/node_modules/webpack/lib/NormalModule.js:718:11)
    at /Users/dowideit.sven/src/rockset/k6/node_modules/webpack/lib/NormalModule.js:778:5 {
  opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
  library: 'digital envelope routines',
  reason: 'unsupported',
  code: 'ERR_OSSL_EVP_UNSUPPORTED'
}

Node.js v18.16.0

Hi , the error is already explored in #33.
@ppcano has promised he will update the repo somewhat and maybe we will even move to rollup #34.
As such I will close this as #33 already tracks the problem described here, which has more to do with the older version of webpack we are using.

@mstoykov nice!
Tho I suspect it is still worthwhile calling out the specific versions of node.js that you guys consider "resaonable" -
do you expect node.js 14.x to work? or just from 18.0 up, etc

Given https://github.com/grafana/k6-template-typescript/blob/34f4a6afde336e3af9aba5302f628e54ae90c4d6/.github/workflows/push.yml#L17-L18  16 seems to be the expected version.
But again this repo needs more then just this updated and if we move to rollup this will likely change


Automate release asset version updates

Automatic upgrade of release assets.

AlDanial/cloc only!
yq -r '."jobs".*."steps"[] | select(."run" | contains("gh api /repos")) | ."run"' .github/workflows/reusable-integrity.yml | grep -o '# v\S\+$'

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
0

Relevant Lines:
3

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Warning: This coverage report may be inaccurate.
This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
0

Relevant Lines:
3

💛  - Coveralls


Add Homebrew formulae in Martin repo

We should try hosting Martin Homebrew formulae in the Martin repo and use the brew tap URL instead of migrating the tap from the Urbica org.
Closes https://github.com/maplibre/martin/issues/616

This was merged, but I have no way to test it

I get Warning: No available formula or cask with the name "maplibre/martin/martin".

@rubenpoppe i sadly have no way of testing homebrew - don't have a mac. Do you know much about that system?  Would love to get some help on this, to help users install it without downloading from the release page

I bought a MacBook weeks ago, maybe any help I can for this?

Seems the homebrew is still broken?  I tried it in #917, but it needs someone familiar with homebrew to add proper tests, and eventually perhaps ensure that it is using latest?  Help wanted :)

Seems to work for me now!

Done in https://github.com/maplibre/homebrew-martin


[SYSTEMML-1891] Prevent 'Script string is blank' exception for Python

@gweidner can you please review this PR ?
@asurve

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/2068/


Docs: Nicer landing page for CLI docs

Use a cards layout on the main landing page instead of the first set of bullet points, to make the CLI docs more shiny.

(see the guides page for an example)

This was fixed in #1103, but the issue wasn't linked. Closing this manually.


fix: set CWD as default path

:tada: This pull request is included in version<EMAIL_ADDRESS>:tada:
The release is available on GitHub release


perf: bump node version in publish workflow

maybe safer to leave it at 16:
https://github.com/Akord-com/akord-js?tab=readme-ov-file#usage


[MSAL for Angular] Sending GET request to undefined url on Auth

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report
[ ] Performance issue
[ ] Feature request
[ ] Documentation issue or request
[ ] Other... Please describe:

Browser:

[x] Chrome version XX
[ ] Firefox version XX
[ ] IE version XX
[ ] Edge version XX
[ ] Safari version XX

Library version

Library version: 0.1.2

Current behavior

I have to clear my local storage and logout of Microsoft to access my angular application after not visiting it for an extended period of time. It was previously working, now I get the following error in chrome console:
GET https:/login.microsoftonline.com//oauth2/v2.0/undefined 404 (NOT FOUND).
For some reason it is sending a GET request with a /undefined attached to the end of it.
Expected behavior

I expect to not have to clear my local storage / logout of of Microsoft to use my app.
Minimal reproduction of the problem with instructions

Wait for access token to expire in browser's local storage. I.e.., when I try to visit the angular app in morning after not using the app overnight. I have to clear my local storage and logout of Microsoft to access the angular application.
More details
I am using the MSAL Angular library with the interceptor. I have it setup to use my app id and have just added the [MsalGuard] to my routes. In my app.module I have
   MsalModule.forRoot({       clientID: environment.clientId,       authority: 'https://login.microsoftonline.com/<App-ID>/',       redirectUri: environment.redirectUri,       cacheLocation : 'localStorage',       protectedResourceMap : [         [environment.apiUrl, [environment.apiScope]]       ]     }),
Nothing got changed in this part of the app.module.ts, and with these settings before it was not producing this error. Does anyone have an idea on where I could start to debug this? It's seems incredibly hard to debug or to track down what the cause of this is.

@jasonnutter I will try to update with more details, but the details seem to be lacking for this issue.

I'm having this exact problem. It happens during silent token renewal after the user hasn't been active on the site awhile. I get a single call to undefined for each msal token renewal frame, followed by a token renewal timeout error.
{_error: "Token renewal operation failed due to timeout", _errorDesc: "Token Renewal Failed", _scopes: ""}

Are you subscribing to the event acquire token failed and then silently try to reacquire?

@jasonnutter I can confirm that we are also getting this same error with @azure/msal-angular 0.1.4-beta.2. Our repro steps are exactly the same, whenever somebody accesses the app for the first time in the day, a GET to https://login.microsoftonline.com/[tenant_id]/oauth2/v2.0/undefined is made. The only recovery from that issue is to clear the local/session storage. I haven's tested whether this is resolved in the new msal-angular beta, but for now we're stuck with @azure/msal-angular 0.1.4-beta.2 until the other issue I reported is resolved.
@nfgallimore I did try your workaround, however one needs to filter out certain broadcasts for this to work. For example, when using acquireTokenPopup and then the user closes the popup, the msal:acquireTokenFailure event is fired with an errorCode of user_cancelled. In that case we do not want to redirect. I still need to dig into exactly what type of error needs to be caught to handle only the "undefined" URL error case.

We have determined this is a bug in login.microsoftonline.com, not the library itself. The service team is triaging now.

it looks like a blocker, any ETA when it might be solved

@kemins Not at this time, unfortunately. I'll follow up when I know more.

In our case, we found that blocking 3rd party was the cause of the issue. After enabling 3rd party cookies, everything behaved as expected.
Not sure why that results in the bad authorize url.

@developerWolf Good to know. Third-party cookies are required regardless, as the library relies on them when acquiring tokens.

I believe this has been fixed, please let us know if not.
Closing.


chore(all) jest 29 upgrade

Jest 29 upgrade to align with tools-suite

includes better support for ESM modules.

What is the reason for the update? I don't mind it, but the PR should contain some information about why we are upgrading.

Added description

Thanks all


Filter bug on v.3.2

Describe the bug
Adding a text filter on a column is including cells that don't have that string.
To Reproduce

On text column click on "Text filter"
Add a search string. In the case it was "19-"
Add a text facet or clustering to see unique values.

Current Results
Things like "O", "ONE AND ONLY" and "GARDASIL" were included in results.
Expected behavior
Only cells with values like "19-2", "19-TWO", "POLICE ---- 19-2" to be returned
Screenshots

Desktop (please complete the following information):

OS: Windows 7
Browser Version: Chrome 76
JRE Version 1.8.0_151

**OpenRefine version 3.2
Datasets
Montreal film permits

@robroc that is likely due to the fact that you are using the records mode and not the rows mode. In the records mode, only one cell within the record is required to match the filter, so if the records contains any other non-matching cells, these will also be visible in the results (and therefore included in the TITLE facet).
https://librarycarpentry.org/lc-open-refine/03-working-with-data/index.html

@wetneb Yes, that solved the issue. It also solved the issue of single items appearing in the clustering interface (instead of multiple items to be clustered). I don't remember seeing this in past versions, even in records mode.


up and download of file shares

The shared folder logic in the gateway storageprovider was not allowing file up and downloads for single file shares. We now check if the reference is actually a file to determine if up / download should be allowed.
fixes: https://github.com/owncloud/product/issues/205 and as a result likely https://github.com/owncloud/product/issues/229, https://github.com/owncloud/product/issues/208 and probably more.
cc @phil-davis

This scenario is still failing: https://cloud.drone.io/cs3org/reva/2544/5/7
  Background:                                                                     # /drone/src/tmp/testrunner/tests/acceptance/features/apiShareManagement/acceptSharesToSharesFolder.feature:7
    Given using OCS API version "1"                                               # FeatureContext::usingOcsApiVersion()
    And using new DAV path                                                        # FeatureContext::usingOldOrNewDavPath()
    And these users have been created with default attributes and skeleton files: # FeatureContext::theseUsersHaveBeenCreated()
      | username |
      | Alice    |
      | Brian    |

  Scenario: When accepting a share of a file, the received file is accessible                                                # /drone/src/tmp/testrunner/tests/acceptance/features/apiShareManagement/acceptSharesToSharesFolder.feature:15
    Given the administrator has set the default folder for received shares to "Shares"                                       # OccContext::theAdministratorHasSetTheDefaultFolderForReceivedSharesTo()
    And parameter "shareapi_auto_accept_share" of app "core" has been set to "no"                                            # AppConfigurationContext::serverParameterHasBeenSetTo()
    And user "Alice" has shared file "/textfile0.txt" with user "Brian"                                                      # FeatureContext::userHasSharedFileWithUserUsingTheSharingApi()
    When user "Brian" accepts share "/textfile0.txt" offered by user "Alice" using the sharing API                           # FeatureContext::userReactsToShareOfferedBy()
    Then the content of file "/Shares/textfile0.txt" for user "Brian" should be "ownCloud test text file 0" plus end-of-line # FeatureContext::contentOfFileForUserShouldBePlusEndOfLine()
      The downloaded content was expected to be 'ownCloud test text file 0
      ', but actually is ''.
      Failed asserting that two strings are equal.
      --- Expected
      +++ Actual
      @@ @@
      -'ownCloud test text file 0\n
      -'
      +''

I was hoping that at least that one would be a newly-passing test.
I will make those sort of Then steps emit more information when they fail - it would be nice to report the HTTP status (and anything else "interesting" in the response.

https://github.com/cs3org/reva/pull/1171#issuecomment-692779128
The download gets a 404 - not found ???

hm the reva logs show 404s but not for a textfile0.txt. In fact textfile0.txt is nowhere to be found ...
oh wait that is an upload ... and if it is using tus the context is disconnected and we have no logs aaaaarrggg. @labkode @ishank011 we really need to reinitialize the logger for the upload context. See https://github.com/cs3org/reva/pull/923
or could we pass a logger / context to all service New() functions so we can attach a logger to the context that is used?

@phil-davis ok, so the problem with the ocis driver is that the skeleton files are not available and they cannot be easily put sumewhere on disk, because the layout is deconstructed. We'll have to use the cs3 api to upload the skeleton.
That is why the test gets a 404 because it cannot share the file.

is this PR for fixing ocis storage only or does it fixes file sharing for other storages as well ?

id fixes acessing single share files for all storage drivers. but I don't know why the tests in reva fail and I am still figuring out how to run the testsuite locally...

The CI currently runs with "owncloud" storage and fails. IMO the skeleton files are correctly "simulated" in that case.
https://github.com/owncloud/core/blob/master/tests/acceptance/features/bootstrap/Provisioning.php manuallyAddSkeletonFiles copies them "underneath" into the storage of the user.

yep, found the readme to run the testsuite locally ... investigating

I used the debugger to halt the requests ... the shadow_folder really  is not populated when accepting the share ... so something is going wrong earlier

@phil-davis https://github.com/cs3org/reva/pull/1170/commits/1af8f4bff42204ccab0a5d9cfeda66f9dbc733ad :crossed_fingers:

@phil-davis this one seems to have gotten better: https://cloud.drone.io/cs3org/reva/2555/4/7
@issue-ocis-thumbnails-191 @skipOnOcis-EOS-Storage @issue-ocis-reva-308 | 271s
-- | --
1316 | Scenario: download previews of other users files                                                                           # /drone/src/tests/acceptance/features/apiOcisSpecific/apiWebdavPreviews-previews.feature:74 | 271s
1317 | Given user "Brian" has been created with default attributes and without skeleton files                                   # FeatureContext::userHasBeenCreatedWithDefaultAttributesAndWithoutSkeletonFiles() | 271s
1318 | And user "Alice" has uploaded file "filesForUpload/lorem.txt" to "/parent.txt"                                           # FeatureContext::userHasUploadedAFileTo() | 271s
1319 | When user "Brian" downloads the preview of "/parent.txt" of "Alice" with width "32" and height "32" using the WebDAV API # FeatureContext::downloadPreviewOfOtherUser() | 271s
1320 | Then the HTTP status code should be "500"                                                                                # FeatureContext::thenTheHTTPStatusCodeShouldBe() | 271s
1321 | HTTP status code 200 is not the expected value 500 | 271s
1322 | Failed asserting that 200 matches expected '500'.

this one seems to have gotten better

Yes, the local API tests are bug-demo scenarios. We expect that when a bug is fixed, then the bug-demo scenario will fail. The process is:

adjust the bug-demo scenario to make it pass
if it is now demonstrating correct behavior, look for the matching scenario in core (there should be one in the matching core feature).
confirm that the core scenario is now passing (it should have been reported in the CI as an unexpected success)
remove the now-passing core scenario from expected-failures
delete the local bug-demo scenario (it no longer demonstrates a bug - it is just a duplicate of a core scenario)

If we had software that already worked, we would not have to try and track all this :)

oohohhhhhh and see what we got:
runsh: Total unexpected passed scenarios throughout the test run:
--
13825 | apiShareManagement/acceptSharesToSharesFolder.feature:15 apiShareManagement/acceptSharesToSharesFolder.feature:22 apiShareToSharesManagementBasic/excludeGroupFromReceivingSharesToSharesFolder.feature:115 apiShareToSharesManagementBasic/excludeGroupFromReceivingSharesToSharesFolder.feature:116 apiShareToSharesManagementBasic/excludeGroupFromReceivingSharesToSharesFolder.feature:153 apiShareToSharesManagementBasic/excludeGroupFromReceivingSharesToSharesFolder.feature:154

fixes https://github.com/owncloud/product/issues/208

apiShareManagement/acceptSharesToSharesFolder.feature:15
apiShareManagement/acceptSharesToSharesFolder.feature:22

Those are a real win - they are the genuine happy-path workflow for ordinary share-accept-download.

that Scenario: download previews of other users files now gives a 200 is worrying ... because it allows access to another users files ... I updated https://github.com/owncloud/ocis-reva/issues/308 but it might be limited to eos ...

@phil-davis did I adjust the tests correctly in https://github.com/cs3org/reva/pull/1170/commits/b0dfd59ee72909b4c9c4b4c2b8aeb26651bb1276 ?

@phil-davis did I adjust the tests correctly in b0dfd59 ?

Yes, looks good. More tests are passing. The preview problem with '500' status is "improved" - anything is better than 500 ! Now the bug-demo scenario shows the "real" issue - a user can download a preview of another user's file, but they should get some 4nn unauthorised status response.

@butonic the code looks good, but this would allow all operations for sharing single files, which we currently don't allow because EOS doesn't support it. Should we allow it @labkode?

@butonic @ishank011 the code can be merged given that this is an optional configuration at the storage provider and is set to false as default. Specially when not all filesystem support it.

@labkode that sounds good. @butonic can you add a config option to allow this change? With a default value of false.

@labkode @ishank011 hm,

at the storage provider

or at the gateway storage provider?
The code checks if the share node exists and is of type file, otherwise it prevents up or download. so this really only allows up and download for referenced files. I don't see the necessity for a config option. If the storage driver does not alllow creating shares on individual shares ... that is an implementation detail of the driver. And it might differ between multiple storage providers. The gateway should be agnostic about this.

@butonic The storage driver doesn't control which files can be shared, that is determined by the share provider and so we have no mechanism in place to stop creating shares for single files for any storage drivers. Because of this, we also can't prevent uploads/downloads at the driver level because we don't know where it's coming from. So it'd have to be checked at the gateway storage provider only.

@ishank011 @labkode I still don't get what you want me to make configurable.
https://github.com/cs3org/reva/pull/1170/files#diff-87a927da7f0ac3d45225420a90987c21R154 is for InitiateFileDownload
https://github.com/cs3org/reva/pull/1170/files#diff-87a927da7f0ac3d45225420a90987c21R396 if for InitiateFileUpload
both are in the gateway storage provider.
do you want me to add a new enable_single_file_shares config to the real storage provider?
or a now config option to the gateway storage provider? the code changes does not  affect the eos driver. shared files will not show up in the Shares folder ...

@ishank011 btw eos does support acls on files: https://github.com/cern-eos/eos/blob/master/doc/configuration/permission.rst#acls

For efficiency, file-level ACLs should only be used sparingly, in favour of directory-level ACLs.

We could add a flag to the eosdriver to disable setting shares on files...

@butonic I've been brainstorming with @ishank011 of what could be done on EOS to support ACL on files and is doable, we'll prototype implementation after finishing etag propagation for the shared folder.
For filesystems not supporting file-based sharing, we need a capability to be queried for the storage provider to avoid showing the share icon in the WebUI.
@butonic rebase, and when is green I'll merge.

@labkode @ishank011 rebased.
Regarding ACLs: CS3 grants != ACLs ... very much so. I realized the following difference when implementing permissions checks for the owncloud driver, which uses extended attributes to store permissions. IIRC we discussed the following scenario at the last workshop:
Given 'einstein' shared folder '/users/einstein/projects/top secret' with 'marie'
And 'marie' accepted the share and mounted it at '/users/marie/Shares/top secret'
And 'marie' created a new folder '/users/marie/Shares/top secret/assets'
When 'einstein' moves '/users/einstein/projects/top secret/assets' to '/users/einstein/top secret assets'
Then 'marie' cannot access '/users/einstein/top secret assets'

Or in short: the assets folder is moved outside of a shared folder.
With ACLs that is not true, because ACLs are propagated down the tree when they are set. This might lead to O(n) writes, where n is the number of children in the tree.
Grants work differently. They are only applied to a single node (file or folder). To check the grant permissions all parent nodes of the path need to be checked. This creates O(n) reads / stat calls, where n is the number of path segments.
Regardless of the storage, an integration with OS level ACLs will always lead to O(n) writes when renaming a shared folder. And for grants we need to implement expiry eg as an extended attribute anyway.
For the owncloud and ocis driver we will go with a pure grant implementation, accepting the O(n) reads tradeoff, because reads can be scaled better than writes. We already have an ACL integration with samba and already see the write bottleneck there when renaming folder. So for us ACL integration is only an additional option.
I plan to change the extended attribute prefix for shares we use to store this from user.oc.acl. to user.oc.grant. and make the difference between acls and grants more clear.
this acl could also be used to check permissions, which would be a different eos driver that has its own permissions checking ... or you know someone to add it to eos ;-)
let me know if we should discuss this in chat.

@butonic CI

@butonic apologies for the delay in replying; this slipped out of my radar.

AFAICT the storage driver should return a not implemented error when trying to persist a grant on a file. because only the storage driver knows what it supports. it is also responsible for not showing the sharing / grant permissions when stating or listing files.

Yes, this should ideally be the case and I was wondering why we don't do that, but the latest EOS actually supports that. I was working with an older version where setting an attribute on a file meant actually setting it on its parent folder, which would lead to huge security risks. So we weren't in the favour of this change without a flag to disable it. But now it looks good to merge. The only issue is that the ACLs are not persisted across versions, which, as @labkode mentioned, we would work on next.

For the owncloud and ocis driver we will go with a pure grant implementation, accepting the O(n) reads tradeoff, because reads can be scaled better than writes. We already have an ACL integration with samba and already see the write bottleneck there when renaming folder. So for us ACL integration is only an additional option.

Makes a lot of sense. Maybe we can also discuss about this change in EOS because the reads overhead would definitely be a lot less than that of writes, but I'm pretty sure that there would be some reason behind that; would be good to know what that is.

But thinking back on this, how often do the writes take place as compared to the reads? A GET or a PROPFIND at the webdav endpoint involves 3-4 stat calls per resource, whereas the writes would occur only when creating new shares or renames. It would be good to run some benchmarks but I think the frequency of read operations makes up quite well for the lower overhead.

the CI failure seems to come from a difference in the XML output of
curl 'https://demo.owncloud.com/ocs/v2.php/apps/files_sharing/api/v1/sharees?search=sharee&itemType=file' -u demo:demo
<?xml version="1.0"?>
<ocs>
 <meta>
  <status>ok</status>
  <statuscode>200</statuscode>
  <message>OK</message>
  <totalitems></totalitems>
  <itemsperpage></itemsperpage>
 </meta>
 <data>
  <exact>
   <users/>
   <groups/>
   <remotes/>
  </exact>
  <users/>
  <groups/>
  <remotes/>
 </data>
</ocs>

vs
curl 'http://localhost:20080/ocs/v2.php/apps/files_sharing/api/v1/sharees?search=sharee&itemType=file' -u einstein:relativity
<?xml version="1.0" encoding="UTF-8"?>
<ocs><meta><status>ok</status><statuscode>200</statuscode><message>OK</message></meta><data><exact></exact></data></ocs>

reva does not show the empty elements when rendering xml (it does when rendering as json), which is exactly what the testsuite says:
Examples:
--
3498 | \| ocs-api-version \| ocs-status \| http-status \|
3499 | \| 1               \| 100        \| 200         \|
3500 | ShareesContext::getArrayOfShareesResponded The sharees response does not have key 'users'
3501 | Failed asserting that an array has the key 'users'.
3502 | \| 2               \| 200        \| 200         \|
3503 | ShareesContext::getArrayOfShareesResponded The sharees response does not have key 'users'
3504 | Failed asserting that an array has the key 'users'.

@phil-davis when was this added to the testsuite ... I don't see where my PR would touch related code. I did remove a few expected test failures, ...
oh wait ...
it is not failing because of those ... it is failing because we actually fixed stuff:
runsh: Total unexpected passed scenarios throughout the test run: | 463s
-- | --
3733 | apiShareReshareToShares3/reShareUpdate.feature:42 apiShareReshareToShares3/reShareUpdate.feature:43 apiShareReshareToShares3/reShareUpdate.feature:76 apiShareReshareToShares3/reShareUpdate.feature:77 apiShareReshareToShares3/reShareUpdate.feature:93 apiShareReshareToShares3/reShareUpdate.feature:94

getting rid of those ...

hm the resharing tests fail again with https://github.com/cs3org/reva/pull/1202 ...


Feature request: datastore delete using key only

Current code expect an entity and then does entity.key

@timanovsky Have you tried deleting by passing the key? It should work.

Dataset can indeed handle both, but not Transaction.
Best regards,
Alexey Timanovsky.
On 26 April 2016 at 17:37, Chris Smith<EMAIL_ADDRESS>wrote:

@timanovsky https://github.com/timanovsky Have you tried deleting by
passing the key? It should work.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-ruby/issues/633#issuecomment-214766371

@timanovsky Thanks for the clarification. I see the issue now. I agree that Transaction should support the same API as Dataset.

This is a bug, I should accept either an Entity or a Key. Thanks for letting us know about it.


Support commands with hidden options

When an option has hidden=True set, Option.get_help_record returns None. Simply filter these out to avoid a later failure when trying to unpack helptext.
I wanted to add a test for this, but there doesn't seem to be a framework for it.
You can check it on any command where hidden=True on an option.
The issue which crops up without this is here:
https://github.com/click-contrib/click-man/blob/3e5a73bda22a8a6ae2e169385173723a11b17409/click_man/man.py#L91
Unpacking that tuple fails if the value is None.

This is a duplicate of https://github.com/click-contrib/click-man/pull/62 that was merged recently.

:shrug: or #62 is a dup of this.
The current sense I get is that this project is defunct (no releases in several years), but it's great prior art for writing your own thing.


Refactor usage of Person

Description
Replace instances of Person in the codebase to use Contact, Client and Vendor.

[x] commons
[ ] logic
[ ] model
[ ] storage
[x] ui
[ ] resources

For tests, comment out Person specific test cases for now since we have not finished implementing behaviour for Contact, Client and Vendor. Add // TODO comments so that we can revisit the test cases at a later milestone.

I don't think we need to update any other "Person" instances in the ui folder


Snorkel Save Linear LR Schedule Model

Description of proposed changes
In Pytorch, we can define a custom LamdaLR function by passing a lambda function. However, Python3 doesn't handle saving lambda's easily. One solution is to change pickle to dill, but may break other features. This breaks saving our model in Snorkel as the original implementation creates a lambda for the LR schedule.
One solution is to create a callable class/object that replaces the lambda (like here).
Test plan
I tested this by running on our existing LabelModel and get the same metric results after training and am able to save the model.
Checklist
Need help on these? Just ask!

[ ] I have read the CONTRIBUTING document.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] I have run tox -e complex and/or tox -e spark if appropriate.
[ ] All new and existing tests passed.

OK. I don't quite understand. Not sure why we need to pickle the LR at all to use the model. But if so, this looks fine.


Optimization build functions

Automated two-layer and three-layer bundling into separate files given the a list of modules for each page of the app.

Something like -
var builder = new require('systemjs-builder')(config);

builder.optimize({
  page1: ['page1-module'],
  page2: ['page2-module', 'another-module']
}).then(function(trees, bundles) {
});

See here for a simple optimize function following the above API I wrote for zygo.

@guybedford When you say three-layer bundling do you refer to have a third list of modules like:
  page1: ['page1-module'],
  page2: ['page2-module', 'another-module']
  page3: ['page3-module', 'another-module', 'yet-another-module']

So that there is an optimisation problem when you try to find the intersects between the trees of the pages? I am still really keen to get into this. Really try sort out my terminology problem.

@Bubblyworld perhaps you can help explain this area?

@guybedford @interactionist Sure, sorry about the delay!
The way I understand three layered bundling is as follows. As you wrote, you have a list of pages and their traced dependencies. Some dependencies might be unique to a page, some might be shared between a few of the pages and others might be shared by all of the pages.
The idea is to create three layers of bundles:

A single common bundle, containing the dependencies shared by all pages.
A layer of bundles containing dependencies unique to each page.
A layer of bundles containing dependencies shared by some of the pages.

The sample implementation I posted above does this the naive way, by (efficiently) taking every subset of pages and creating a bundle for dependencies shared within each subset. This could potentially result in a large number of bundles, which kinda defeats the point of doing bundling at all. (X_x)
So the problem is to split up the modules into bundles for the best possible user experience. What this means and whether it's even worth solving I'm not sure. I also might be completely missing the mark, so take this with a pinch of salt!

Any progress on this? It's the only thing that is stopping me from moving to jspm ;)

@Bubblyworld
Is the dependency object a flat object like:
{
  'page1' : ['module1', 'module2', 'module3'],
  'page2' : ['module1', 'module4']
  'page3' : ['module2', 'module3']
}

So that given a page key we get a one-dimensional array of deps?

@interactionist Yes, that's what I assume in that example I posted before.

I have been working on a bundling solution using systemjs-builder too, and came up against the same problem identified by @Bubblyworld ... How do you split the middle layer of bundles up so that you don't potentially end up with a zillion bundles each containing one or two modules?
One solution would be to hand-craft the bundles trying to logically group modules by area and potentially offloading some modules that appear in most entry point trees into the common bundle. However that would increase exponentially both in difficulty and being a massive drag to maintain as the project grows so it's not really viable.
For our app we have a number of possible entry points but some are more likely than others and regardless of entry point a typical user journey would most likely load most of the entry points anyway. By weighting each entry point and prioritising the most commonly used ones we can bundle with no repetition and still be quite lean in the code loaded initially.
var entryPoints = [
    'src/popularEntryPoint',      // 100% efficiency - no unnecessary code it loaded for this entry point.
    'src/anotherEntryPoint',      // If any of the models in this dependency tree are also in popularEntryPoint's dependency tree, just load that bundle as well. Since it's the most popular entry point it's quite likely the user will need these files soon anyway.
    'src/yetAnotherEntryPoint',   // Maybe this tree will share modules with popularEntryPoint or perhaps anotherEntryPoint as well.
    'src/leastPopularEntryPoint'  // Etc.
];

This way you may be loading code you don't need for entry points further down the list but the chances are that since those parts of the site / app are more popular the user will need the files soon anyway.
Our optimisation function takes an array like the one above and an integer specifying how many bundles you want. That way you don't necessarily load 187 files when entering via the least popular entry point in a site. Limit it to, say, 8 bundles and it will perform the following steps:

If applicable, create a bundle of common modules to all entry points (so long as the number of bundles requested > 2).
Create bundles for each entry point in priority order until (number of bundles created) == (bundles requested)-1.
Take any remaining entry points and bundle them all into the final bundle.

The function also takes a switch to turn on analysis which spits out some interesting numbers regarding file sizes and entry point code loading efficiency.
What do you think of this method of optimisation? Worth including as an option, even if not the default optimisation function?
I'm happy to take on this feature if no one is already tackling it...

I agree, there is a lot more data that might be useful for a custom optimization function that what we are considering processing at the moment. Do you think the optimize promise should a) resolve with an object full of trees, or b) just indicate that the bundling process is complete and the files have been written? A) would give the user more of an opportunity to further analyse and break down the output but b) would avoid extra code for what I'd assume to be the common use case.

I've written up some notes in pseudo code form to try and clarify the API around builder.optimize and the custom optimization function.
Would you mind taking a look and letting me know what you think?
// Because you can't rely on the order when enumerating properties on an object,
// the best way to weight them without specifying weight as an attribute somewhere
// is in an array.
builder.optimize([
    ['src/module1', 'src/module2'],
    'src/module3',
    'src/module4'
]).then(...);

// By making the elements of the array objects we can specify a name for the bundle
// which can be used in the file name. It also opens the way for other metadata
// about this bundle that might be useful in custom optimization functions
builder.optimize([
    {
        entryPoint: ['src/module1', 'src/module2'],
        name: "bundle1"
    },
    {
        entryPoint: 'src/module3',
        name: "module3Bundle"
    },
    {
        entryPoint: 'src/module4',
        name: "myBundle"
    }
]).then(...);

// Could give a choice of syntax as the first is easier if you are happy with default bundle names?

builder.optimize(entryPoints, optimizationFunction).then(...);

/**
 * Optional, user-defined function to process the tree data generated from the entry points
 * @param  {[{entryPoint:String|[String], loadRecord:Object, gZippedFilesize:Number, tree:Object[, name:String]}]} data
 *         Array of objects containing the entry points passed into builder.optimize,  any other
 *         data passed into builder.optimize including the bundle name (if provided) and the resulting
 *         tree from a trace operation on the entry point(s).
 * @return {Promise|{bundleName:Tree, ...} Promise to be resolved with an object of Trees or
 *         an object of Trees, with each tree representing a bundle to be written.
 */
function optimizationFunction(data) {

    // <clever manuipulation of trees happens here>

    return {
        bundle1: <tree>,
        module3Bundle: <tree>,
        myBundle: <tree>
    }

    // Note the bundle names don't have to be used in the returned data structure if the optimization
    // algorithm calls for a different practice. Also, if no names are provided default names will be
    // used here (bundle0, bundle1, bundle2, etc).
}

builder.optimize(entryPoints).then(function(data) {
    // data is the same data as the data returned from the optimizationFunction
    // (this is where I get a bit unsure about things - should we require the developer to take this
    // step or should it be an implicit part of the optimize process? If so that's more data to pass
    // in up front. I'm leaning towards this step since they might like to inspect / fiddle with the
    // output before writing it).
    builder.writeTrees(data, { minify: true, sourceMaps: true, uglify: false });

});

@markstickley thanks for this. I think the way we would do weighting and entry point priorities is through a special options object that gets passed through to the optimizer. This way any custom optimization options can be passed through without us needing to characterize everything.
Also it would be nice to have a single optimize call that does all the writing to bundles, with a memory mode just like we have for the current build.
Here is an adjusted API I'd suggest based on the points mentioned:
builder.optimizeBuild({
  optimizationFunction: require('custom-optimizer'),
  entryPoints: {name: modules},
  optimizationOptions: { ...custom optimize variables, entrypoint priorities etc... }
  outPath: 'out/folder', // optional, if not set returns source as memory compilation
  sourceMaps: true
  // etc other options
})
.then(function(bundles) {
  //bundles[0] = {
  //  name: 'bundle-1', // name of the bundle in the outpath
  //  entryPoints: [], // entry points this bundle is loaded for
  //  modules: [], // modules in this bundle
  //  source: // provided if in memory mode and no outPath given
  //}
});

The builder then does a full trace of all the modules of all the entry points.
It passes the direct trace object through to the optimize function, so it has direct access to all the metadata on the load records. This way it can do source size checking etc, completely on its own.
function optimize(entryPoints, trace, optimizationOptions) {
  // entryPoints is exactly as above
  // optimizationOptions is exactly as above
  // trace is an object hash containing all traced dependencies of the modules
  // of the entry points and all their dependencies.
  // trace['some/module'] = load
  // where load is the direct trace load record from the loader
}

Let me know how that looks to you!

It is need aslo to include CSS output source in memory as well (current build desn't have this).

Ideally we could just turn bundling into a linear constraint problem and use something like https://github.com/slightlyoff/cassowary.js?

Ideally we could just turn bundling into a linear constraint problem

What function are you going to optimize (minimize probably)? And what constrains to use?

@guybedford Yes that looks very sane. I'll set about implementing that, cheers!
Just one thing, I'm not 100% clear what the return value of the optimization function should look like. Are you able to clarify?
Thanks :)

I guess it would be -
bundles = [{
  name: 'bundle-1', // name of the bundle in the outpath
  entryPoints: [], // entry points this bundle is loaded for
  modules: [], // modules in this bundle
}]

Alternatively we could returned an object keyed by bundle name rather?

I think I like the name attribute better as if the names are omitted it will use a default naming convention. While they will still all have names in end, keying off those names is less useful if you don't know what they will be in advance.
Since entryPoints is not optional, I think perhaps it should come outside the opts variable in builder.optimizeBuild. Perhaps the same for optimizationFunction? The problem with a default optimization function is if it requires extra information (like entry point weighting) it makes using optimizeBuild much less intuitive. If we require an optimization function to be specified then there can be extra instructions based on which one you choose.

All good points! So do you think something like -
builder.optimizeBuild(entryPoints, require('custom-optimizer'), {
  // not handled by builder -
  // custom configuration options needed by the optimizer get passed through
  entrypointPriorities: {},
  // builder-specific options:
  outPath: 'out/folder', // optional, if not set returns source as memory compilation
  sourceMaps: true
  // etc other options
})

Looks good to me :)

Couple of questions.
Is there a standard way of generating systemjs config or a standard place to write it to? Currently in the implementation I've been working on it writes bundles and depCache config to config.js in the same output folder as the rest of the built files.
Also, the work I've done has the option of outputting metadata about each bundle and the bundling process as a whole. In the current specification for the optimization function's return value, data pertaining to the process as a whole isn't supported. In order to avoid complexity do you think it would be better to return an instance of an object that conforms to an API? That way you could have (for example) returnValue.getNames(), returnValue.getBundles(), returnValue.getBundle(name), returnValue.getEntryPoints(name), returnValue.getModules(name) as well as any custom extensions like returnValue.getBundleEfficiency(name) etc.
Thanks!

@markstickley yes there is currently only one config which is changed from dev to production. Having a special "production config" being created is an issue being tracked at https://github.com/systemjs/builder/issues/67.
In terms of the additional metadata, we could possibly allow the individual bundle entries to be overloaded? Or does that not cover everything you need?

Ah #67 looks like what we need here to create/update the config file. Since all the config code is in jspm-cli, is it OK for builder to depend on that? I would think that config specific to bundles would most likely be production config. I think that means this task has as a dependency on #67 would you agree?
As for the additional metadata, overloading the bundle entries is fine but if there is metadata about the build as a whole (speed, efficiency, size etc) I'm wondering where that should go. Technically you could break down those figures and add them to each bundle to be totted up later but it's more of a convenience thing.

@markstickley was trying to create an API that didn't have configuration assumptions as a dependency. If we want to see the config as part of the system, then perhaps we should reconsider the API itself. I can go either way on this though - on the one hand its about specifying how users would want the config to be generated for pages, on the other hand its about characterizing the minimal optimization build. I was trying to take the shortest definite path first in specifying a comprehensive optimization that can be generalized as part of a great config process later on to avoid having to do all that spec work upfront though - because chances are I'd get it wrong. Note that builder is designed to be wrapped in other tools while it doesn't have the high-level APIs, so there's no problem with that either.
At the same time, perhaps creating this arbitrary API boundary is the wrong framing as well, and we should be looking at more comprehensive solutions. I'm not sure.
Perhaps we just make bundles a property of the output object, with other properties allowed then? Since this is what the optimization function returns it could then choose to add its own extra properties.

@guybedford Hmm it's a difficult balance. Here's what I suggest:

By default, write config.js to the same folder as the bundles. If config.js already exists then append it, otherwise create it.
Extend the output object to also include a config property which has depCache and bundles arrays that can be added to an existing config.js
Provide another function in the builder API which can be bound with paths for bundles and config.js and used as the callback in optimizeBuild().then();

builder.optimizeBuild(entryPoints, require('custom-optimizer')).
then(builder.writeOptimizedBuild.bind(builder, {
    bundlesPath: 'built',
    'configPath': '.'
});

That way the relative path for the bundles can be calculated as well
System.bundles['relative/path/to/bundle'] = ["pages/page1","utils/myUtil"];

Just for clarity I wrote out the spec again taking into account the discussions. Please should if anything looks wrong.
var entryPoints = {
  bundle1Name: 'path/to/bundle1/entryPoint',
  bundle2Name: ['path/to/bundle2/entryPoint', 'path/to/anotherEntryPoint/includedIn/bundle2'],
  bundle3Name: ['path/to/bundle3/entryPoint']
};
// Note: entryPoints can also be an array if you are happy to use default bundle names

builder.optimizeBuild(entryPoints, require('custom-optimizer'), {
  // builder-specific options:
  outPath: 'out/folder', // optional, if not set returns source as memory compilation
  sourceMaps: true,
  uglify: true,
  minify: true,
  // etc other options
  // custom configuration options not handled by builder but needed by the optimizer get passed through:
  entrypointPriorities: {},
  // etc...
}).
then(function(optimizedData) {
  console.dir(optimizedData);
/*
{
  bundles: [
    {
      name: 'bundle1Name',
      entryPoints: 'path/to/bundle1/entryPoint',
      modules: ['path/to/dependency1','path/to/dependency2'], // modules in this bundle
      source: '...' // provided if in memory mode and no outPath given
    },
    ...
  ],
  config: {
    depCache: {
      'path/to/bundle1/entryPoint': ['path/to/dependency1','path/to/dependency2']
    },
    bundles: {
      'out/folder/bundle1Name': ['path/to/bundle1/entryPoint','path/to/dependency1','path/to/dependency2']
    }
  }
}
*/
});

/**
 * Optional, user-defined function to process the tree data generated from the entry points
 * @param {{String|String[]}|[String|String[]]} entryPoints Object or array of entry points (strings or arrays of strings)
 * @param {Object} trace Full trace data mapping to the entry points
 * @param {Object} optimizationOptions Custom variables that can be required or optional for this optimization function
 * @return {Promise|{bundleName:Tree, ...} Promise to be resolved with an object of Trees or
 *         an object of Trees, with each tree representing a bundle to be written.
 */
function optimizationFunction(entryPoints, trace, optimizationOptions) {

  // <clever manuipulation of trees happens here>

  return {
    bundles: [{
      name: 'bundle-1', // name of the bundle in the outpath
      entryPoints: [], // entry points this bundle is loaded for
      modules: [], // modules in this bundle
    },
    ...
    ],
    config: {
      depCache: {
        ...
      },
      bundles: {
        ...
      }
    }
  };

  // Note the bundle names don't have to be used in the returned data structure if the optimization
  // algorithm calls for a different practice. Also, if no names are provided default names should be
  // used here (bundle0, bundle1, bundle2, etc).
}

@markstickley yes that looks correct. Note that the config output is not from the optimization function but added within SystemJS itself. I'm actually looking at adding a similar output for the single-file build so would be good to align with that. See https://github.com/systemjs/builder/issues/67#issuecomment-77568917.

Also note the entry points don't refer to bundle names but actual entry point names.

Entry points should never be arrays I don't think - it's against the definition of an entry point.

I guess it could make sense though.

I've created a public gist for the spec so it's easier to edit and keep track of changes! https://gist.github.com/markstickley/c1bc6663cbe36bc0d46e
I agree that entry points shouldn't be arrays but I was trying to incorporate the original suggestion at the top of this thread. I'll remove support for that.
The entry points don't actually refer to bundle names but the name of the entry point can be used when creating the bundle filenames. But they don't have to be so I'll change that to make it clearer.
Thanks!

@markstickley perfect thanks for summarizing, it looks like we're on the same page with that.

Quick update - things are going well! Just writing some tests.
Should I include a default optimization function within systemjs-builder or should all optimization functions live somewhere else externally? If internally, should I make it possible to run without specifying a function (so it uses a default) or should it be specified explicitly?
Finally, I'm still not sure what to do with the SystemJS config. Currently it's being written to the same location as the bundles but I realise this may not be ideal. Any advice on this? Cheers!

@markstickley that is awesome to hear!
What sort of algorithm are you planning to have the default optimization function follow?
For the config, I wouldn't do anything here. baseURL is dist-specific. After that the only map configuration one needs in production is the configuration needed specifically for dynamic loading (I've changed bundles to use normalized IDs only to allow this in the next release). That is, the bundles config object only. Perhaps this could even be inlined into the page-specific bundles given that we know page ordering.

@markstickley thanks so much for your efforts on this to date. I think an optimization may be best handled through a wrapper around this project, and it maintains responsibilities much better anyway.

Use http://stealjs.com!