Build failure: mission-center

Steps To Reproduce
Steps to reproduce the behavior:

build mission center

Build log
nix-shell -p mission-center this derivation will be built:   /nix/store/jhr74v8dyzm38la54p98p521ap30ifhq-mission-center-0.6.2.drv building '/nix/store/jhr74v8dyzm38la54p98p521ap30ifhq-mission-center-0.6.2.drv'... Using versionCheckHook Running phase: unpackPhase unpacking source archive /nix/store/pdzy121sbx8zgpqjfbgjd12sx02yxb08-source source root is source Executing cargoSetupPostUnpackHook Finished cargoSetupPostUnpackHook Running phase: patchPhase substituteInPlace called without any files to operate on (files must come before options!) error: builder for '/nix/store/jhr74v8dyzm38la54p98p521ap30ifhq-mission-center-0.6.2.drv' failed with exit code 1;        last 8 log lines:        > Using versionCheckHook        > Running phase: unpackPhase        > unpacking source archive /nix/store/pdzy121sbx8zgpqjfbgjd12sx02yxb08-source        > source root is source        > Executing cargoSetupPostUnpackHook        > Finished cargoSetupPostUnpackHook        > Running phase: patchPhase        > substituteInPlace called without any files to operate on (files must come before options!)

Build Log
error: builder for '/nix/store/jhr74v8dyzm38la54p98p521ap30ifhq-mission-center-0.6.2.drv' failed with exit code 1;
       last 8 log lines:
       > Using versionCheckHook
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/pdzy121sbx8zgpqjfbgjd12sx02yxb08-source
       > source root is source
       > Executing cargoSetupPostUnpackHook
       > Finished cargoSetupPostUnpackHook
       > Running phase: patchPhase
       > substituteInPlace called without any files to operate on (files must come before options!)

Additional context
I am on unstable branch
Metadata

system: "x86_64-linux"
host os: Linux 6.12.0, NixOS, 25.05 (Warbler), 25.05beta710087.23e89b7da85c

Notify maintainers
@GaetanLepage
@getchoo

Note for maintainers: Please tag this issue in your PR.

Add a :+1: reaction to issues you find important.

This is fixed on master. The issue is that the nixos-unstable channel is lagging a bit further than usual.
Let's just wait for the fix to land there.
https://nixpkgs-tracker.ocfox.me/?pr=357219


Release Google.Cloud.Parallelstore.V1Beta version 1.0.0-beta01

Changes in this release:
Initial release.

Created release for Google.Cloud.Parallelstore.V1Beta-1.0.0-beta01


Add HTML comment support to template processor

Currently the template processor will still process any commands it finds in
While insertion into the dom still means any html will not be actioned the process will still perform actions on {{}} and @- commands.

Best to do this in the initial template loader. That way we reduce repetition of replacement for multi use templates


Fix document scrollTop

When replacing context.html remove the old html code and the page is reduced in height after adding a new block with the new html code scroll will not mix into the original position

You can close it was fixed by the author in https://github.com/yiisoft/jquery-pjax/commit/d3e5570e8e62393d9e250c3946a0511ceae647bc


[cloud_firestore] does not like underscore in iOS bundle id

Following instructions in:
Flutter Firebase codelab
I named my project friendly_chat.
The problem is that Dart packages need to use the snake_case naming convention. When the iOS bundle name is generated, it uses the project name unchanged. Copying and pasting that into the Firebase console won't work because Firebase does not accept the underscore.

@stevemessick
The issue at https://github.com/flutter/flutter/issues/13518 has been closed and moved here. Future collaboration on this issue will be done here.

Hi
does this proposed solution  work for you?

does this proposed solution work for you?

Hard to say because I'm not sure what the proposal is. Some detail about what is changing might help.

does this proposed solution work for you?

Hard to say because I don't know what the proposal is. Some detail about what changes are being made might help.

@stevemessick as identified in the original issue, _ are not allowed in bundle IDs. Replacing them with dashes should resolve your issue.


LIMIT REACHED: The number of request query parameters to exclude for the target rule exceeds the limit of 10

Hi Team,
Hi Team,
We've used the CA module to establish preconfigured_waf_config_exclusions and encountered an issue while incorporating request_params into our exclusion list. An error emerges, stating:
Error: Error updating SecurityPolicy "poc": googleapi: Error 400: Invalid value for field 'resource.preconfiguredWafConfig': '**  "exclusions": [**    "targetRuleSet": "rce-v33-stable",    "targetRuleIds": ["owasp-crs-v030301-id...'. The number of request query parameters to exclude for the target rce-v33-stable:owasp-crs-v030301-id932150-rce exceeds the limit of 10., invalid
Our configurations are entirely managed through Terraform. Is there a method to either increase this limit or find a workaround so we can add the necessary exclusions to the rule?
Any advice would be greatly valued.
Regards,
Swapnil Pawar

@imrannayer - Tagging you becoz seen your inputs on the issues. Thanks!

@spawar-apex can u plz post your code here?

@imrannayer  - Thank you for your prompt Response. Here is the code:

Replaced values with test

`preconfigured_waf_config_exclusions = {
    # This is a list of general exclusions that apply to most of the rules in the RCE ruleset
    rce_general_exclusions = {
      target_rule_set = "rce-v33-stable"
      target_rule_ids = [
        "owasp-crs-v030301-id932100-rce", "owasp-crs-v030301-id932105-rce", "owasp-crs-v030301-id932110-rce",
        "owasp-crs-v030301-id932115-rce", "owasp-crs-v030301-id932150-rce", "owasp-crs-v030301-id932160-rce"
      ]
      request_query_param = [
        {
          operator = "EQUALS"
          value    = "test"
        },
        {
          operator = "EQUALS"
          value    = "test"
        },
        {
          operator = "EQUALS"
          value    = "test"
        },
        {
          operator = "CONTAINS"
          value    = "test"
        },
        {
          operator = "CONTAINS"
          value    = "test"
        },
        {
          operator = "CONTAINS"
          value    = "test"
        },
        {
          operator = "CONTAINS"
          value    = "test"
        },
        {
          operator = "CONTAINS"
          value    = "test"
        },
        {
          operator = "ENDS_WITH"
          value    = "test"
        }
      ]
      request_cookie = [
        {
          operator = "CONTAINS"
          value    = "test"
        },
        {
          operator = "CONTAINS"
          value    = "test"
        },
        {
          operator = "CONTAINS"
          value    = "test"
        }
      ]
    }

    # Exclusion for owasp-crs-v030301-id932140-rce
    rce_exclusion_1 = {
      target_rule_set = "rce-v33-stable"
      target_rule_ids = ["owasp-crs-v030301-id932140-rce"]
      request_query_param = [
        {
          operator = "EQUALS"
          value    = "test"
        }
      ]
    }

    # Exclusion for owasp-crs-v030301-id932150-rce
    rce_exclusion_2 = {
      target_rule_set = "rce-v33-stable"
      target_rule_ids = ["owasp-crs-v030301-id932150-rce"]
      request_query_param = [
        {
          operator = "CONTAINS"
          value    = "test"
        },
        {
          operator = "EQUALS"
          value    = "test"
        }
      ]
    }

  }
}`

For the rule owasp-crs-v030301-id932150-rce, we would like add more exclusions but when trying to add, hitting the limit.

@imrannayer - You may notice multiple rce_exclusion created becoz we though multiple exclusion wont limit the request_query_param limit

@spawar-apex have u tried doing the same using console or CLI? Just want to check if this is a limit in service or TF

@imrannayer - I have tried using Console and I did check the limit/quota of cloud armor. There is nothing short of this. This is happening only when creating exception via terraform.

@imrannayer - Sorry for the confusion. I did check again and Looks like Its provider level configuration limit. I tried to check it in console and I got the same error message.

@imrannayer - That's correct. I did reach out to GCP support and working with them actively to resolve this issue.


A crop to the input video defaults NVEnc to 1920x520p

Hi,
I am not sure how to solve this problem but it seems that if an input video is cropped in staxrip to 1920x800 the Nvidia Encoder will handle it as 1920x520p. This is referring to the latest version of staxrip 1.3.2 and NVENC SDK 6.0. Also, is there a documentation of what command lines or options we could use with NVENC ?
I noticed that if I do not crop the input video, the encoder will handle it as 1080p :
NVEnc 2.00?3 (x64), using NVENC API v6.0
OS Version         Windows 10 (x64)
CPU                Intel Core i7-4930K @ 3.40GHz (6C/12T)
GPU                #0: GeForce GTX TITAN X (24 EU) @ 1076 MHz (362.00)
Input Buffers      CUDA, 32 frames
Input Info         Avisynth 2.60(yv12)->nv12 [AVX], 1920x1080, 24000/1001 fps
Output Info        H.265/HEVC main
1920x1080p 1:1 23.976fps (24000/1001fps)

please post the full log to paste.bin, make it expire after 30 days


Crash when using get_gravity from _process while running physics on separate thread

Tested versions
Reproducible in: 4.3-dev [acde2a81f]
System information
Windows 11 (10.0.22631)
Issue description
The new get_gravity method that was added in #84640 relies on body_get_direct_state, which is not accessible outside of _physics_process and _integrate_forces when using the physics/*d/run_on_separate_thread project setting.
As such, it ends up throwing an error here:
https://github.com/godotengine/godot/blob/63d6bda8e95ac992da74e84b2f3be62f3d85190b/servers/physics_3d/godot_physics_server_3d.cpp#L924
... and subsequently crashes due to a nullptr access in get_gravity.
Steps to reproduce

Run the MRP
Observe crash

Minimal reproduction project (MRP)
GravityCrash.zip

I'm on it

I appreciate that an error is preferable to a crash, but this maybe wasn't the fix I had hoped for. :)
I'll make a new issue for the error I guess.

This matches the rest of the code, wasn't going to add an additional error originally
But you can't avoid an error entirely

Fair enough. Maybe creating an issue wasn't the right tool to use here. Ideally I wouldn't have fumbled the review of #84640 and this could've been discussed prior to merging this change, but alas.
I agree that your fix was the pragmatic fix in this case, and lines up with how get_contact_count does it, and an error is absolutely better than no error. My issue was more with the usage of body_get_direct_state to begin with, but I guess that's a broader topic that's maybe better suited for elsewhere, as you point out.
(Also, I didn't mean for my previous comment to come off as snarky as it did. I appreciate the quick fix, of course.)

Thank you! Making a report was the appropriate thing to make sure the crash was resolved, that shouldn't have been left untended to, but a broader improvement would be good to discuss too :)


Edit talk comment

For #361 and https://joindin.jira.com/browse/JOINDIN-686

Can one of the admins verify this patch?

Ok to test

ok to test

Want me to get those curly braces in the new line?

Please :-)

Thanks @runz0rd


Should bicycle=yes be really stated as cycleway?

I got an email stating that this path is not a cycleway it is only highway=track|path and bicycle=yes

Sorry, the path in question is highway=residential. I am the person who signalled the problem to Peter
It looks as if you are interpreting the combinations highway=unclassified|residential|tertiary|secondary|primary AND bicycle=yes as presence of a cycleway.
The combination highway=path|track AND bicycle=yes is, correctly, not considered as cycleway

Although @ratrun seems to have understood your issue I'm not that fast this time :)
So you think all is fine except for highway=residential and bicycle=yes where it should not be a cycleway?

It doesn't look as if @voschix has received your question. The best way to see what was meant is to open the link above, study the directions and in parallel open the area in JOSM to study the tags of the choosen ways. It is valid for all kind oy ways, even for highway=motorway with bicycle=yes. Here the instruction classified such a segement as cycleway - which was not the intended behaviour when I implemented this pretty long ago.

What about highway=path, bicycle=designated? This is preset tag for tagging combined and segregated cycleways and footways in JOSM. osm wiki.

@buma: highway=path, bicycle=designated should be classified as cycleway.
See following test: https://github.com/graphhopper/graphhopper/blob/master/core/src/test/java/com/graphhopper/routing/util/AbstractBikeFlagEncoderTester.java#L306

@karussell: Sorry I had overlooked your implied question.
My original warning was (which I had sent via e-mail):
"When a way is tagged with any of
highway=residential|unclassified|tertiary|secondary|primary
AND
bicycle=yes
the turn-by-turn instructions refer to a cycleway (in Italian “pista ciclabile”) on that road. This is not correct. bicycle=yes only states that bicycles are allowed on that road, but it is not a cycleway
Note that the combinations
highway=track|path
AND
bicycle=yes
are correctly not interpreted as cycleway."
(slightly reworded)

Thanks again! @ratrun PR is merged now!


Update Azure NIC and NSG modules to support sovereign cloud environments

This PR updates the network interface and network security group testing modules to support Azure sovereign clouds (Government, Germany, China, Azure Stack). This behavior is controlled by setting the AZURE_ENVIRONMENT environment variable to the appropriate environment name (as covered in examples/azure/client_factory.md).
Closes #813

Closed this PR and will reopen from newer fork.


Is there a way to specify the batch size to reduce VRAM?

Running large models on large files require a lot of memory, but other libraries like whisper and whispex allow you to pass a batch_size to the CLI. Is there an equivalent option for stable-ts?

Batch transcription is not supported on the original Whisper models (i.e. batch size is always 1) so there is no batch size parameter to control for reducing memory usage.
However, you can reduce memory usage on the Hugging Face models by specifying batch_size because the default, batch_size=24, uses significantly more memory than the original models.
https://github.com/jianfch/stable-ts/blob/6d066308ed5a3328a69006d3a7d4496315736c0f/stable_whisper/whisper_word_level/hf_whisper.py#L186
The best way to reduce memory usage is to use a distilled or/and quantized large model from Faster-Whisper or Hugging Face.


topology: Consume v6 alphas and build Penta demo

Once the official alpha is announced, it should be pulled in to user feedback's v6 branch.
Also blocked until https://github.com/patternfly/react-topology/issues/133 is completed
The v6 branch should also be updated to include all changes made to the main branch since the v6 branch was created.
Once the code merges and a v6-alpha of this extension is available, it should be pulled into the v6 branch on org.
Update per @jenny-s51 on 4/23/24:
Will require splitting up this effort:

[ ] #181
[ ] Apply tokens to styles

Closed by https://github.com/patternfly/react-topology/pull/201


Remove not accurate claim

The issue
Readme contains a misleading claim "Windows is a first-class citizen, in our world.".
Since claim itself or use of pipenv overall in Windows is not covered by docs by any means.
Removal of this claim should help save time for newcomers to match alternatives with Windows support.
Claim should be restored once docs properly covering work on Windows and claim can be considered as accurate.
The fix
Removes a claim for better times.

Sorry, but I can't get what this PR is aiming to do. We are doing a lot of jobs getting Pipenv work on the Windows platform. And I personally also mainly use Windows + Powershell. If you encounter any problem with Pipenv on Windows, please file an issue. I would like to close this PR now.

@frostming pipenv docs has poor(only found path to cache on Windows is given and nothing else) coverage for Windows.
Here is some links to pages for alternative package system that covers work with Windows, that pipenv is lacking.
https://python-poetry.org/docs/#windows-powershell-install-instructions
https://python-poetry.org/docs/basic-usage/#activating-the-virtual-environment
While pipenv docs provide commands for unix based oses only.
Having this in mind claim "Windows is a first-class citizen, in our world" sounds inaccurate.
Do you understand my point of view?
Overall I appreciate the work you doing!

@frostming my change should save time for people checking docs in search of instructions for Windows.
For claim "Windows is a first-class citizen, in our world" it is expected to have best user experience + docs coverage across all OS-es.
From what I see - Linux is a first-class citizen for pipenv.

I use pipenv on Windows and have not had platform specific problems. If you find situations where pipenv works on Linux but not Windows, report them, and we’ll work on fixes. It does not help to throw claims without explanation.


User friendly name of Downloaded Templates  Volumes and ISOs

ISSUE TYPE

Enhancement Request

COMPONENT NAME

UI

CLOUDSTACK VERSION

4.17.1

CONFIGURATION

OS / ENVIRONMENT

N/A
SUMMARY

STEPS TO REPRODUCE

End user clicks on Download Template, ISO or Volume

EXPECTED RESULTS

Filename in the format "resourceName.ext" for templates, isos or volumes being downloaded.

ACTUAL RESULTS

Filename in the format "UUID.ext" for templates, isos or volumes being downloaded.

@alexandremattioli I've added to the GSOC2023 ideas, can you please present the idea in a paragraph for better understanding of potential contributors?

@nvazquez
Currently in ACS when an end-user clicks on the download button for a tempalte, iso or volume, the file name is in the form UUID.ext.  It would be extremely useful for the file name to be resourceName.ext instead.

@nvazquez Currently in ACS when an end-user clicks on the download button for a tempalte, iso or volume, the file name is in the form UUID.ext. It would be extremely useful for the file name to be resourceName.ext instead.

@alexandremattioli, I think what @nvazquez means is to edit the issue description to read as a full functional spec, so it will be easy to assess for interns if they want to work on it.

Copy @DaanHoogland

Yes, @DaanHoogland is correct, sorry if it wasn't clear

Hi @AayushSaini101,
Thanks for reaching us, I would recommend a good way to start is the Hackerbook: https://github.com/shapeblue/hackerbook, and for further reading on the core concepts, the documentation: http://docs.cloudstack.apache.org/en/latest/
Please let us know if you have any issues

Thanks, @nvazquez  for the reply. Over the last two days. I am trying to set up the project, but unable to execute the server management.  I am getting build failure error and cannot able to execute the server. I am using rhel8 and project  SDK is 8 and also I have tried by using setting SDk to 11.

@AayushSaini101 can you pull the latest code form the main branch? Otherwise, you can try pulling the code from the latest release tag: <IP_ADDRESS>. Also, it seems like you are building using the noredist profile, is this correct? If so, please get the dependencies from this repo: https://github.com/shapeblue/cloudstack-nonoss into your deps directory and install them via the script before building

@nvazquez Thanks above changes works. Now I am facing an issue regarding the DB Setup while executing the mvn -P developer -pl developer -Ddeploydb

@shwstppr Thanks, I am trying this. Also replacing MariaDB to Mysql in my local

@shwstppr @nvazquez Still I cannot able to run it is showing

I have already followed the above instructions in the handbook. I am using MariaDB

Thanks but still is showing Access Denied. Granting to root@localhost

@nvazquez Thanks But it contains only one file db.properties

@AayushSaini101 ok, are you able to access mysql using the password set on the file by mysql -u root -p?

@nvazquez  Yes

@AayushSaini101 can you post the content of the db.properties file hiding the passwords? I suspect the root password field may be empty

@nvazquez

@AayushSaini101 thanks for sharing, can you change these first lines to:
db.cloud.username=cloud
db.cloud.password=cloud
db.root.password=<YOUR_ROOT_PASSWORD>

CloudStack will create the cloud user (you can choose another password for it) via root user

Thanks @nvazquez Finally Resolved. How to switch this

@AayushSaini101 great! Now you only need to bring up the UI to interact with the management server. Please check: https://github.com/shapeblue/hackerbook/blob/main/2-dev.md#building-cloudstack last section:
To bring up the Cloudstack Ui

    $ cd /path/to/cloudstack/ui
    $ sudo apt install nodejs npm
    $ npm install
    $ npm run serve

Make sure to set CS_URL=http://localhost:8080/client on .env.local file on ui.

You should be able to run the management server on http://localhost:5050/

Thanks working on it

Great, well done @AayushSaini101 - let us know if you have further issues or questions

@nvazquez  I am unable to add Register ISO, it is showing unable to find image store. I tried to find this in docs, but able to find.

@AayushSaini101 looks like you don’t have a secondary storage defined, which hypervisor are you using? You can consider using the simulator that allows you to create a test simulated zone from a script

@nvazquez Can you please tell me where script is located also can you please refer to some of the videos to understand cloudstack much better. In Installation guide there are many videos which are outdated

@AayushSaini101 you may need to build the project and setup databases for simulator as explained on: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30748425#content/view/30748425
After that, with your management server running, you can deploy a test zone with this command:
python ./tools/marvin/marvin/deployDataCenter.py -i setup/dev/advanced.cfg

(may need to run this before: pip install tools/marvin/dist/Marvin-*.tar.gz)

fixed in #9252


No Data from some hosts

Hello. After upgrading Zabbix 5 to 6.0.2 I'm getting an Error 500 when getting data from some hosts ("Filesystem /: Space utilization" from "Template OS Linux by Zabbix Agent"). I had configured Zabbix Data source through API, changed from server to browser and it didn't work. Also changed to Direct DB connection and it was the same.

Zabbix 6 introduced api changes that are the cause of this.
duplicate of #1396

Hi! Can you post grafana logs here? I cannot replicate this one. Did you update plugin as well?


Pass cwd into extension provided tasks

Issue Type: Bug
Created a task that used "option: { "cwd": "${workspaceFolder}/examples" }" to run the npm start at the child repo, then typed Ctrl+Shift+B to launch it.
It seems that VSCode attemped to check if the task configuration exist in the package.json, but it ignored the cwd option.
For example, I have a task like this:
        {
            "type": "npm",
            "script": "start",
            "group": {
                "kind": "build",
                "isDefault": true
            },
            "options": {
                "cwd": "${workspaceFolder}/examples"
            }
        }

Then, if both the ${workspaceFolder}/package.json and the ${workspaceFolder}/examples/package.json has the npm start script, everything works well. But if I removed the npm start script from the ${workspaceFolder}/package.json, it would failed to launch the task, just output like error: npm task check not support the configured task.

 All attributes set in tasks.json on an extension provided task are added on top of the task after the task as been created by the extension. Because of this, the npm start task that the npm extension provides doesn't know that you've set the cwd. The feature request here is to pass cwd into the extension provided task.
You can do this now if you make a task of "type": "script" and run npm start from there.

Thanks, but if  the npm start task that the npm extension provides doesn't know that you've set the cwd, why it works correctly after we add any same name script into the ${workspaceFolder}/package.json? The problem just happend in checking stage, and if the check passed, then everything works well (the script witch launched in fact is in the child examples project), which indicate that it's not an problem of extension ...doesn't know ...the cwd.

You can see how the tasks are created by the extension here: https://github.com/Microsoft/vscode/blob/release/1.34/extensions/npm/src/tasks.ts
Specifically, we only look in the workspace folders: https://github.com/Microsoft/vscode/blob/release/1.34/extensions/npm/src/tasks.ts#L119

As the task in fact could launched correctly, the checking should be passed too. Now it can't, so I wonder that maybe it is really unreasonable. If the checking prevented the correctly task executing, I prefer to close the checking.

I think if you correctly provide the "path" in your task then it should work. The path should be the relative path do the package.json with the script, and should not include the basename of package.json


[BUG] Publishing Github release doesn't work in v0.5.2

Environment
changelogen: 0.5.2
node: 18.14.1
browser: Arc v0.95.1 (37921)
Reproduction
Run: changelogen --release --push
package.json
{
  "changelog": {
    "repo": {
      "provider": "github"
    }
  }
}

Describe the bug
The auto publish of the release with Github not works anymore with v0.5.2
A new window of the Browser is open, stay open and the release is not published.
Additional context
The urls generated for commits and compare version looks like:

https://undefined/undefined/compare/v0.0.8...v0.0.9

I think the bug is due to this error
Check it here: https://github.com/zadigetvoltaire/nuxt-gtm/blob/main/CHANGELOG.md (v0.0.9)
Logs
> changelogen --release --push

ℹ Generating changelog for v0.0.8...main                                                                                15:27:31
ℹ Bumping version from 0.0.8 to 0.0.9 (minor)                                                                                15:27:31
ℹ Updating /.../CHANGELOG.md                                                                 15:27:31
ℹ Followup in the browser to manually create the release.

I already set the repository url:
{
  "repository": {
    "type": "git",
    "url": "https://github.com/zadigetvoltaire/nuxt-gtm.git"
  },
}

I downgraded the version to 0.5.1 and it's working well.
If I delete this configuration, the release synchronization with Github will not be done automatically when I run changelogen --release --push ?
{
  "changelog": {
    "repo": {
      "provider": "github"
    }
  }
}


Set specific path for file-loader to look for assets

I'm using file-loader along with extract-loader and html-loader to process images within some generated static HTML (Hugo is the generator).
The generated HTML and assets are all under a public/ folder, the structure is:
webpack.config.js
public/
    index.html
    images/
    some-path/
    |   index.html
    |   another-path/
    |   |    index.html

etc.
All images are referenced absolutely as /images/image.png from all HTML files. Some are at a path like /images/foo/bar/image.png.
I have successfully gotten Webpack to process all of the HTML files, but since the images all come with absolute paths it doesn't know where to find them in the context of my project root (at least with my current config). I have also successfully processed a single HTML file and its images by altering the image paths of that single file, so I know the config works in that regard. The image paths were hashed in HTML and the image filename was also hashed.
Is there a way I can tell file-loader and Webpack where to find these images, without manually altering their path in the HTML?
Here is my current config:
           test: /\.(png|jpg|svg)$/,
                use: [{
                    loader: "file-loader?outputPath=images/",
                    options:  {
                        limit: 10000,
                        emitFile: true,
                        useRelativePath: true
                    }
                }]
            },
            {
                test: /\.html$/,
                use: [
                    "file-loader?name=[name].[ext]",
                    "extract-loader",
                    "html-loader?" + JSON.stringify({
                        attrs: ["img:src"]
                    })
                ]
            }

output: {
        path: path.resolve(__dirname + '/public/'),
        publicPath: "./public/",
        filename: '[name].js'
    }

The entry is an object  glob containing all of the HTML under the public/ folder.
I'm using file-loader v0.11.1 and Webpack v3

You can set publicPath: '' and at runtime set __webpack_public_path__ = 'path/to/your/assets'.
Note that currently you have to downgrade to file-loader 0.10.1 to make this work.

Hi @AndreKR, I'm having the same issue as @jewbetcha. I followed your advice and added publicpath to my webpack.config and set the public path inside my entry point file. Sadly, this wont work cause the CSS imports with the URL link inside them will load earlier then setting the webpack_public_path.
My stackoverflow post (initially for url-loader) https://stackoverflow.com/questions/46001976/webpack-url-loader-dynamic-public-path
Thanks in advance,
Tim

First double-check please that you are really using file-loader 0.10.1 and not something newer.
Then, what's in your ./src/admin.planningview.js? If is contains something like
__webpack_public_path__ = 'path/to/your/assets';
require('base.css');

then it should work.

@jewbetcha If you figured out the solution to this would you mind posting it in a comment please? Running into a similar issue where the home page is in root and image paths are ./img/ but subdirectory/routes have the same when they need to be ../img.

@poorpaddy I haven't figured it out 😭 I'm either missing something obvious, or this is just a thing Webpack isn't made for (working with assets that aren't built with JS frameworks)

Problem still exists? Somebody can create minimum example repo and description?

Yes. The problem still exists, I got the same issue if I try to load images form subfolders. When i use ./img/ everything is fine but if I got an .html file in a subfolder to load images I need to use this ../img/ and it doesn't work.

I managed to fix this by just using ./ as my root. However it is far from a perfect solution. In the end I ended up no longer bundling images in webpack and now serve them from CDN. It actually drastically improved build times and load times since I served the images thru a cache.

I have the same problem and I recognize that webpack stores the images in the folder which are set in the config.output.path1 + path2, where path2 is in '[path2][name].[ext]' when setting the option for file-loader (path2 is the original location of the images).
The browser tries to retrieve the images from another location such as app_url/path2, however. I have tried different configuration for webpack but failed. I don't know if someone could do that.
My temporary solution is copying the images to the necessary folder after running webpack. I hope to hear a solution from someone else so that the browser retrieves the images where webpack stores in.

Please create minimum reproducible test repo

@mimi149 I ended up using other way for handle images:
https://github.com/webpack-contrib/copy-webpack-plugin
so I only use file-loader for fonts


Maven jib:dockerBuild in Google Cloud Build fails

Description of the issue:
Attempting to build and deploy a container from within Google Cloud Build.
jib:build wants me to authenticate within the same project where GCB and GCR run. It feels wrong to attempt to add this so I am attempting to use jib:dockerBuild.
  - name: 'gcr.io/cloud-builders/mvn'
    id: MVN_COMPILE
    args: ['compile','jib:dockerBuild', '-Dmaven.test.skip=true', '-Ddocker.image.prefix=gcr.io/$PROJECT_ID', '-s', '.mvn/settings.xml']

images:
  - 'gcr.io/$PROJECT_ID/my-company-project'

But I am getting an error that Google Cloud Build does not have a docker daemon running. I can run both the jib:build and jib:dockerBuild locally, but need the CI to work as a trigger in Google Cloud Builder.
Steps to reproduce:
Simply using Google cloud build via triggers or gcloud builds submit .
Environment:
jib-maven-plugin Configuration:
            <plugin>
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>jib-maven-plugin</artifactId>
                <version>1.0.0</version>
            </plugin>

Log output:
Step #3 - "MVN_COMPILE": [ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.0.0:dockerBuild (default-cli) on project my-company-project: Build to Docker daemon failed, perhaps you should make sure Docker is installed and you have correct privileges to run it -> [Help 1]

Hi @dgvigil,
We are aware of this particular issue.
FYI, GCB recently dropped Docker from gcr.io/cloud-builders/mvn. In fact, now the image has essentially become the official mvn Docker Hub image. jib:dockerBuild no longer works.
In fact, jib:build should just work and push the built image to the GCR. There's an internal bug open on GCB to make this work. The bug is that /builder/home/.docker/config.json, which holds the GCR credentials, is not being copied/linked into /root/.docker/config.json on GCB.
That said, the current ugly workaround you can use is to link that file yourself. Something like
steps:
- name: gcr.io/cloud-builders/mvn
  dir: /root
  entrypoint: bash
  args:
  - -c
  - # Links the Docker config to /root/.docker/config.json so that Jib picks it up.
    # Note that this is only a temporary workaround.
    # See https://github.com/GoogleContainerTools/jib/pull/1479.
    |
    mkdir .docker &&
    ln -s /builder/home/.docker/config.json .docker/config.json
  volumes:
  - name: user.home
    path: /root

- name: gcr.io/cloud-builders/mvn
    id: MVN_COMPILE
    args: ['compile','jib:dockerBuild', '-Dmaven.test.skip=true', '-Ddocker.image.prefix=gcr.io/$PROJECT_ID', '-s', '.mvn/settings.xml']
  volumes:
  - name: user.home
    path: /root

(I know this is not pretty. Should be unnecessary once the bug is fixed.)

Another option is to have docker-credential-gcr on $PATH by whatever means:
- name: gcr.io/cloud-builders/mvn
  entrypoint: bash
  args:
  - -c
  - |
    export PATH=$$PATH:. &&
    curl -fsSL https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_amd64-1.5.0.tar.gz | tar zxv &&
    chmod +x docker-credential-gcr &&
    mvn -B compile jib:build -Dimage=gcr.io/$PROJECT_ID/chanseok-test

or
- name: gcr.io/cloud-builders/mvn
  dir: /usr/local/bin
  entrypoint: bash
  args:
  - -c
  - |
    curl -fsSL https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_amd64-1.5.0.tar.gz | tar zxv &&
    chmod +x docker-credential-gcr
  volumes:
  - name: usr.local.bin
    path: /usr/local/bin
- name: gcr.io/cloud-builders/mvn
  args: ['-B', 'compile', 'jib:build', '-Dimage=gcr.io/$PROJECT_ID/chanseok-test']
  volumes:
  - name: usr.local.bin
    path: /usr/local/bin

Thanks for the workarounds @chanseokoh. Appreciate a quick heads-up here once the internal bug is fixed.

I found a simpler solution:
- name: gcr.io/cloud-builders/mvn
  args: ['-Duser.home=$$HOME', 'compile', 'jib:build', ...]

This makes sure that the user.home seen by java/maven is aligned with $HOME.
This might also make the build run faster because maven will stop downloading files under /root, that's inside the docker container, and rather download them under the /builder/home volume.
That also has the advantage of being able to share that folder between multiple steps (I think).

@dgageot in fact we were discouraging setting user.home to /builder/home, because $HOME points to /root in GCB (for example, Maven caches artifacts under /root/.m2) and -Duser.home= changes that JVM-wise. There has been a discussion around what $HOME should actually be on GCB, and because currently it is /root we have been recommending not overriding it.

... changes that JVM-wise.

So for example, I believe -Duser.home=/builder/home will make Maven cache artifacts under /builder/home/.m2. (Correct me if I am wrong.

So, on GCB, $HOME is set to /builder/home. It's only java's user.home that is set too /root for a reason that I'm still investigating.
Also, yes, if you set the user.home to /builder/home it will indeed also download maven artifacts under /builder/home/.m2 which is great because then you are not writing inside the container but in a shared volume that is faster and shared by build steps and could be cached.

Oh, yeah, my mistake about $HOME, sorry. To be fair, I did know $HOME points to /builder/home, which is very confusing actually (I did note this in the internal bug before), because the OS-defined home directory for the user root is still /root. (The Maven runs as root and stores artifacts under /root/.m2.)

I think java is basically reading the user's home directory from /etc/password -> root
GCB is setting the HOME variable to /builder/home when docker running the build step.

https://stackoverflow.com/questions/1503284/java-system-getpropertyuser-home-returns/1508616#1508616

Anyways, I don't really see the risk of changing user.home

This issue will be one of those that the new feature #1902 will fix. Once #1902 gets in, no more extra configuration will be needed for GCB. Will update here once it goes live.

1.5.0 released that can pick up the Google Application Default Credentials on most GCP environments including GCB. In many cases, Jib will work out of the box on GCB by default with no extra configuration.
If anyone sees an issue, follow up on #1903.
@dgvigil @ractive @janwillies @grimrose @Matrix89 @priyawadhwa @dgageot @Alos


Pan PTZ controls ^<>v

Hello,
Is there any way to send pan commands ?
Thanks
Rd/

No. You need to use HTTP to send ptz commands. Nothing to do with RTSP protocol.


Add event_id to web events

Add event_id to snowplow_web_events model. This is useful for joining clean event data to custom contexts.

@jtcohen6 this looks simple enough to me :)
Can you confirm that adding this column doesn't interfere with other columns called event_id in a downstream model?

@drewbanin It works when I run locally. There are no columns named event_id downstream of this model.


calling ROOT via Cxx.jl

I was able to get ROOT.jl (on https://github.com/jpata/ROOT.jl/tree/cxx) with ROOT 6 to work directly through Cxx.jl under Mac OS X without re-compiling the julia binary to load-and-hide ROOT before the julia LLVM is called.
This seems to be possible due to LLVM >=3.6 using versioned symbols as described in https://bugs.freedesktop.org/show_bug.cgi?id=93103. However, ROOT still uses an old LLVM with un-versioned symbols.
julia> versioninfo()
Julia Version 0.5.0-dev+5503
Commit 93ee8dd* (2016-07-19 06:48 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin15.4.0)
  CPU: Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, haswell)

$ root-config --version
6.06/02

Enabling RTTI in Cxx.jl: https://github.com/jpata/Cxx.jl/pull/2, https://github.com/Keno/Cxx.jl/issues/104 (on top of https://github.com/Keno/Cxx.jl/commit/f6bc213b335e3c87a6dd2a08a0846813a259abfe)
ROOT prints the usual error but runs successfully:
$ ~/Documents/julia/julia ~/Documents/.julia/v0.5/ROOT/test/rootcxx.jl
Error in <UnknownClass::InitInterpreter()>: LLVM SYMBOLS ARE EXPOSED TO CLING! This will cause problems; please hide them or dlopen() them after the call to TROOT::InitInterpreter()!

Trying the same under linux (via https://github.com/jpata/docker-julia-root-cc7/blob/master/Dockerfile) fails with a segfault
(gdb) bt
#0  0x00007ffff6d145f7 in raise () from /lib64/libc.so.6
#1  0x00007ffff6d15ce8 in abort () from /lib64/libc.so.6
#2  0x00007ffff6d54327 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff6d5a194 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffddd0081e3 in clang::LangOptions::~LangOptions() () from /root/.julia/v0.5/Cxx/src/../deps/usr/lib/libcxxffi-debug.so
#5  0x00007ffdde068152 in clang::PrintingPolicy::~PrintingPolicy (
    this=0x7ffddaa809a0 <TCling::InspectMembers(TMemberInspector&, void const*, TClass const*, bool)::printPol>, __in_chrg=<optimized out>)
    at /opt/julia/usr/bin/../../usr/include/clang/AST/PrettyPrinter.h:35
#6  0x00007ffff6d17e69 in __run_exit_handlers () from /lib64/libc.so.6
#7  0x00007ffff6d17eb5 in exit () from /lib64/libc.so.6
#8  0x00007ffff6d00b1c in __libc_start_main () from /lib64/libc.so.6
#9  0x00000000004012e9 in _start ()

Here's the docker image to reproduce that
jpata/julia               v0.5_root_cxx       f7e4de6be62a        About an hour ago   5.989 GB

ROOT looks for the following LLVM symbol to detect a clash: which is named a bit differently under OSX
$ nm -a /Users/joosep/Documents/julia/usr/lib/libLLVM-3.7.dylib | grep Pretty
0000000000cb8c10 T _LLVMEnablePrettyStackTrace

and linux
$ nm -a /opt/julia/usr/lib/libLLVM-3.7.so | grep Pretty
00000000004a6b40 T LLVMEnablePrettyStackTrace

The long-term fix would renaming the LLVM symbols when building it for julia: https://github.com/JuliaLang/julia/issues/12644
@oschulz @jpivarski you may be interested in this

@jpata: Very nice! My Cxx.jl-based ROOT bindings (https://github.com/mppmu/ROOTFramework.jl - have a look, it can even run the ROOT GUI) was limited to ROOT-5 until now, because of the LLVM issue. I'll try with ROOT-6 and your patches.

compiling ROOT from master still results in a segfault under linux:
#0  std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (
    this=0x7ffdbbe9cd40 <TCling::InspectMembers(TMemberInspector&, void const*, TClass const*, bool)::printPol+128>, __str=...)
    at /mnt/build/davidlt/gcc530_final/b/BUILD/slc6_amd64_gcc530/external/gcc/5.3.0/gcc-tags_gcc_5_3_0_release-231255/obj/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:617
#1  0x00007ffdbe73059e in clang::LangOptions::LangOptions (
    this=0x7ffdbbe9ccc0 <TCling::InspectMembers(TMemberInspector&, void const*, TClass const*, bool)::printPol>)
    at /mnt/t3nfs01/data01/shome/jpata/tth/julia/usr/bin/../include/clang/Basic/LangOptions.h:48
#2  0x00007ffdba3349ba in TCling::InspectMembers(TMemberInspector&, void const*, TClass const*, bool) ()
   from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_11/external/slc6_amd64_gcc530/lib/libCling.so

@jpata: I tried using ROOT6 via jpata/Cxx.jl#2 (with my ROOTFramework.jl and your example from your DIANA talk), but as soon as the first ROOT class is called, julia exits with this error:
Error in <UnknownClass::InitInterpreter()>: LLVM SYMBOLS ARE EXPOSED TO CLING! This will cause problems; please hide them or dlopen() them after the call to TROOT::InitInterpreter()!
: CommandLine Error: Option 'inline-threshold' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

Any ideas on what's causing the Option 'inline-threshold' registered more than once?

@oschulz I don't really know. I also can't get ROOT6 via Cxx to work under linux, giving me the impression that there is still some fundamental conflict between the two LLVM-s.

I've made some progress.
@oschulz Try the cxx branch now. It will still compile a ROOT-enable julia binary, but it will live in the julia source tree in usr/bin/rjulia and behave in general exactly like the main binary. It seems to work well under OSX and Linux. In the end at CERN we'll probably make a custom compilation that is Cxx and ROOT enabled.

Thanks, @jpata, I'll try!
I know that Keno also was trying some things to get ROOT-6 compatibility in CxxROOT.jl, but I don't know what the status is.

Hm, on further reflection, I'm not so sure it's a good idea for a package to install something to the julia source tree. For example, we use a share julia installation for several people in our group - so that would mean that only the admin can install ROOT.jl - and things like ROOT.jl will warrant frequent updates. Or maybe the rjulia could live in a package of it's own (and wouldn't need to be updated often), separate from Julia wrappers around ROOT classes?

Just an idea: rjulia could be a small separate C++ project, to be built independenty and linked against an existing Julia and ROOT installation - but installed separately. That way, one could also use one Julia installation with several ROOT versions (or the other way around). Or does rjulia need to live next to the julia binary for some reason? If not, it should be enough to install it somewhere in PATH.
Of course I still hope that Keno and the ROOT team will find a way to make Julia and ROOT play nicely together, eventually. :-)

@jpata: I was able to apply and build your patch. However, I had to change
[...] `root-config --cflags --libs --ldflags` $^ -o $@ [...]

to
[...] $^ -o $@ `root-config --cflags --libs --ldflags` [...]

in the Makefile to avoid linker errors. (I'm on Ubuntu. Depending on the linker configuration, linking order can matter nowadays, https://wiki.debian.org/ToolChain/DSOLinking).

Thanks for trying.

CxxROOT.jl is a great idea, I first head about it in December '15 when Keno wrote the code. It aims to replace TCling in ROOT with a Cxx.jl-based interpreter. It needs significant work that is beyond my knowledge. Depending on how modular ROOT is in that part, it may not offer full compatibility and may need significant changes to ROOT. In any case, I don't know how much time @Keno has to work on it in the near future.

Indeed, one would have to think about the packaging. I used to compile the binary in the Pkg directory, but it's somewhat easier (sysimg, JULIA_HOME, access to julia build env etc) to do it directly in the source tree. It wouldn't be a user-installed package on a shared system. One can think to still compile it in the Pkg directory (and later add it to PATH) by propagating -H -J flags, but I'm not sure it's better.

In my view, it's not necessary to re-create all the ROOT class structure and hierarchy on the julia side, like is done in PyROOT. I would have ROOT.jl be just a very thin layer that allows you to call ROOT via @cxx. We can think about offering julia-esque wrappers around some ROOT structures akin to rootpy, but re-coding all the structure of ROOT in julia seems hopeless (unless you do it automatically), especially as you can never offer full syntactic equivalence to C++.

For reference, PyROOT dynamically generates proxies to the corresponding
ROOT classes. It's not just automated, but also just-in-time.
Unfortunately, this magic doesn't seem to work in Julia's PyCall. You can
@pyimport ROOT, but none of the proxies exist. (In Julia 0.4. And even if
it did work, using PyROOT in Julia would not be an efficient way to access
ROOT.)
Jim
On Jul 29, 2016 1:48 AM, "Joosep Pata"<EMAIL_ADDRESS>wrote:
Thanks for trying.

CxxROOT.jl is a great idea, I first head about it in December '15 when
Keno wrote the code. It aims to replace TCling in ROOT with a Cxx.jl-based
interpreter. It needs significant work that is beyond my knowledge.
Depending on how modular ROOT is in that part, it may not offer full
compatibility and may need significant changes to ROOT. In any case, I
don't know how much time @Keno https://github.com/Keno has to work on it
in the near future.

Indeed, one would have to think about the packaging. I used to compile
the binary in the Pkg directory, but it's somewhat easier (sysimg,
JULIA_HOME, access to julia build env etc) to do it directly in the source
tree. It wouldn't be a user-installed package on a shared system. One can
think to still compile it in the Pkg directory (and later add it to PATH)
by propagating -H -J flags, but I'm not sure it's better.

In my view, it's not necessary to re-create all the ROOT class structure
and hierarchy on the julia side, like is done in PyROOT. I would have
ROOT.jl be just a very thin layer that allows you to call ROOT via @cxx. We
can think about offering julia-esque wrappers around some ROOT structures
akin to rootpy, but re-coding all the structure of ROOT in julia seems
hopeless (unless you do it automatically), especially as you can never
offer full syntactic equivalence to C++.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jpata/ROOT.jl/issues/4#issuecomment-236107428, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABxEHxBsqAtjUe2sm2KHyErqJhnML5-Eks5qaaJKgaJpZM4JQo5Q
.

Yep, I've seen the code that does it in PyROOT. PyCall+PyROOT was something I considered a while back, before embarking on ROOT+Clang.jl. There's also a way of directly accessing the ROOT proxies from PyCall using some python trickery, but it's all rather brittle.
To me, reflecting the whole of a complex C++ library in julia or python is bound to be doomed. In particular, the semantics of references, pointers and so on are lost in the PyROOT approach. Until recently, one could not really create any class from the PyROOT side which didn't have explicit "dictionaries" generated, so using functions like func(std::vector<Object>) was impossible.
Given that we have a way of directly calling C++ from julia, I would endeavour to make it as painless as possible. I think providing bespoke julian interfaces on top of TFile, TTree, TH*, ... as much as possible is useful, this is mostly what ROOTFramework.jl does.
If calling some more obscure ROOT functionality from julia is needed, one can still do it using cxx""" //C++ code here """.
The one issue that is killer is still the LLVM incompatibility, I don't have any good ideas about that.

On 29 Jul 2016, at 13:32, Jim Pivarski<EMAIL_ADDRESS>wrote:
For reference, PyROOT dynamically generates proxies to the corresponding
ROOT classes. It's not just automated, but also just-in-time.
Unfortunately, this magic doesn't seem to work in Julia's PyCall. You can
@pyimport ROOT, but none of the proxies exist. (In Julia 0.4. And even if
it did work, using PyROOT in Julia would not be an efficient way to access
ROOT.)
Jim
On Jul 29, 2016 1:48 AM, "Joosep Pata"<EMAIL_ADDRESS>wrote:
Thanks for trying.

CxxROOT.jl is a great idea, I first head about it in December '15 when
Keno wrote the code. It aims to replace TCling in ROOT with a Cxx.jl-based
interpreter. It needs significant work that is beyond my knowledge.
Depending on how modular ROOT is in that part, it may not offer full
compatibility and may need significant changes to ROOT. In any case, I
don't know how much time @Keno https://github.com/Keno has to work on it
in the near future.

Indeed, one would have to think about the packaging. I used to compile
the binary in the Pkg directory, but it's somewhat easier (sysimg,
JULIA_HOME, access to julia build env etc) to do it directly in the source
tree. It wouldn't be a user-installed package on a shared system. One can
think to still compile it in the Pkg directory (and later add it to PATH)
by propagating -H -J flags, but I'm not sure it's better.

In my view, it's not necessary to re-create all the ROOT class structure
and hierarchy on the julia side, like is done in PyROOT. I would have
ROOT.jl be just a very thin layer that allows you to call ROOT via @cxx. We
can think about offering julia-esque wrappers around some ROOT structures
akin to rootpy, but re-coding all the structure of ROOT in julia seems
hopeless (unless you do it automatically), especially as you can never
offer full syntactic equivalence to C++.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jpata/ROOT.jl/issues/4#issuecomment-236107428, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABxEHxBsqAtjUe2sm2KHyErqJhnML5-Eks5qaaJKgaJpZM4JQo5Q
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@jpata: In my view, it's not necessary to re-create all the ROOT class structure and hierarchy on the julia side, like is done in PyROOT. I would have ROOT.jl be just a very thin layer that allows you to call ROOT via @cxx

To me, reflecting the whole of a complex C++ library in julia or python is bound to be doomed. [...]  I think providing bespoke julian interfaces on top of TFile, TTree, TH*, ... as much as possible is useful, this is mostly what ROOTFramework.jl does.

I fully agree, and I like your idea of having one slim package to provide the necessary low-level compatibility support (ROOT.jl), and a separate package (like my current ROOTFramework) to provide Julianic convenience wrapper around frequently used ROOT functionalities (and maybe a Plots.jl backend, one day?). Certainly wrapping all of ROOT is not feasible - nor necessary with Cxx.jl. And the parts that we do want provide Julia wrapper for (like THxx, TFile, TTree) have to be carefully crafted manually, not automatically, to be ideomatic Julia.
Until now I thought we would merge ROOT.jl and ROOTFramework.jl at some point - but I think I prefer your layered approach, Joosep. In that case, I want to find a better name for ROOTFramework.jl though, it was just chosen to not conflict with ROOT.jl, and always considered temporary. :-) I'm open for proposals, anyone got a good idea?

I wouldn't still outrule merging the two packages at some point, I think for users it might be clearer if there is only one ROOT package, providing the base compat layer through Cxx and some limited "nice-to-have" julian wrappers.
The need to recompile the binary etc might go away and then we'd just have a single ROOT package.
IMHO first we've to figure out a strategy of what we wrap and how. rootpy has gone the way of basically manually pythonizing File, Tree, Chain, Hist etc. Their interface quite battle-tested, I would strongly favour taking example from them, e.g.
type Tree
    baseobj::CppValue{:TTree}
   ...
end
function Tree(name=nothing, title=nothing, model=nothing)
#like https://github.com/rootpy/rootpy/blob/master/rootpy/tree/tree.py#L886
end

In ROOT.jl I attempted to autogenerate the hierarchies using Clang.jl, but it will never map properly I think.

The need to recompile the binary etc might go away and then we'd just have a single ROOT package.

Sure, once the LLVM problem has gone away, there will hopefully be no need for two packages. As long as we need to compile stuff to make Julia and ROOT compatible, though, it's probably better to keep that separated from the easy-to-update convenience wrappers.

rootpy has gone the way of basically manually pythonizing File, Tree, Chain, Hist etc. [...] I would strongly favour taking example from them

Well, yes and no. Julia is not Python, and not object oriented. So IMHO we shouldn't follow Python conventions where there is already a different Julia way to to a certain kind of task. We should also take full advantage of multiple dispatch, etc. in the Julia wrappers.
Also, we should strive free the Julia users from some of the historic awkwardness of ROOT (the the ROOT developers want to get away from long-term, too). Most importantly, I'd like to shield them from ROOT's way of having a global "current" file and putting some things into it automatically (TTree creation is a good example). For example, we could use getindex to access TDirectory (and therefore TFile) contents, like mytfile[TTree, "treename] or so.
Good Interface- / API-design is never independent from the language and it's conventions, and in several ways, Julia and Python differ very much. So instead of pythonizing ROOT, I'd like to julianize it. :-)
Also, it may be better not to wrap the CppValues, like in
type Tree
    baseobj::CppValue{:TTree}
   ...
end

but use them directly. It will make it easier to access non-wrapped functionality via @cxx in a natural fashion.

The one issue that is killer is still the LLVM incompatibility, I don't have any good ideas about that.

We're working on it. ROOT is carrying a number of patches that are not acceptable to upstream, so we're trying to work with them to figure out how to clean that up.
As for API design in general, I've strongly advocated not to just do some sort of automatic translation, but let C++ be C++ and have the wrapper packages expose more julian APIs on top of what C++ provides (i.e. integration with julia side abstractions, iterators, trees, arrays, etc.)

the LLVM incompatibility
We're working on it.

Thanks!! I can imagine that it's not a trivial problem at all.

API design in general, I've strongly advocated not to just do some sort of automatic translation, but let C++ be C++ and have the wrapper packages expose more julian APIs on top of what C++ provides

Yes, that's what I've tried to do with ROOTFramework, so far. It's still very much a work in progress (and I probably need to revise some concepts). But my goal is to make it as julian as possible for the wrapped functionality, and to allow users to easily fall back to @cxx for the non-wrapped stuff. As Joosep said, ROOT is just to big to wrap it all. ;-)

@oschulz

getindex is used in HDF5.jl, perhaps we can think if TFile, TDirectory etc map onto that.
For TTree, TChain, I've investigated an AbstractDataFrame interface in ROOTDataFrames.jl

getindex is used in HDF5.jl, perhaps we can think if TFile, TDirectory etc map onto that.

I didn't look into HDF5 in detail yet, but that's certainly a good reason to emulate it. One thing that held me back so far is that ROOT objects already have a name (not a design concept I like, but ...) and are usually written to a file under that name. So mytfile["myhist"] = h would not write h under it's current name. Though maybe that's not such a bad thing.

For TTree, TChain, I've investigated an AbstractDataFrame interface in ROOTDataFrames.jl

Yes, I saw that. I think something like that should be an add-on package (like it is now). ROOTPlots.j (Plot.jl integration) would be another candidate. I'm considering to directly include things like THxx<->StatsBase.Histogram, though, because histograms are so central to ROOT and one can hardly avoid having StatsBase installed anyhow.

On the topic of histograms and StatsBase:
https://github.com/jpata/ROOTHistograms.jl
https://github.com/jpata/Histograms.jl

If this e-mail thread is a brainstorming session, here's an idea:

Don't dynamically link into ROOT from Julia, which has problems with
incompatible dependencies (LLVM). Instead, run it as a separate process.

Julia has an equivalent of Python's "multiprocessing," but deeply built
into the language. When you @spawn an expression, it must serialize all the
objects and send them through a pipe, and it must have some protocol for
worker Julias to talk to the master Julia.
Here's the idea: implement that worker protocol in C++ in ROOT. When ROOT
gets serialized Julia objects (a derivative of HDF5), it should unpack them
as C++ objects, possibly using Cling to make the types and to compile and
execute the desired expression. For user familiarity, it should use the
same Julia data <--> C++ data mapping as Cxx.jl.
The @spawnroot macro would work a little differently than @spawn, in that
the expression it gets has to be turned into a C++ AST or string for ROOT
to evaluate. Since Julia macros already act on ASTs, there's a lot of
flexibility here. For instance, Julia ASTs can look like objects and
methods in dot-notation, such as '@spawnroot tfile.ls()'. If you like, you
can retain C++'s distinctions between '.' and '->' by faking a pointer:
transform '@spawnroot tfile.pointer.ls()' into 'tfile->ls()'. (Or use some
shorter word like 'to'.)
This separate-process interface wouldn't be as fast as direct linkage, but
it would be more flexible. Objects would not live in the same memory space,
which can help to avoid death due to segmentation fault. In fact, if a ROOT
process dies, Julia can spawn another one with less disruption to the user.
Remote ROOT processes can be accessed on clusters with ROOT installed and
no Julia installed (typical of physics installations, right now at least).
Although each invocation of @spawnroot has a lot of overhead and shouldn't
be used in a tight loop, it can contain a tight loop that runs as fast as
C++ (because after conversion it is just C++). This is still faster than
PyROOT because PyROOT does transformations every time it sees a dot, every
time it steps through a loop.
In some systems (Spark and other tools from the Hadoop ecosystem come to
mind), this separation of work into independent processes is not seen as a
concession or a hack, but as a way of isolating and recovering from failure
(because operating systems are more robust than most process runtimes).
And then Julianic interfaces can be built on top of this, like rootpy on
PyROOT. (Is "Julianic" really the word? Not "Julian" or "Jubilant?")
-- Jim
On Fri, Jul 29, 2016 at 9:37 AM, Joosep Pata<EMAIL_ADDRESS>wrote:

I wouldn't still outrule merging the two packages at some point, I think
for users it might be clearer if there is only one ROOT package, providing
the base compat layer through Cxx and some limited "nice-to-have" julian
wrappers.
The need to recompile the binary etc might go away and then we'd just have
a single ROOT package.
IMHO first we've to figure out a strategy of what we wrap and how. rootpy
https://github.com/rootpy/rootpy/tree/master/rootpy has gone the way of
basically manually pythonizing File, Tree, Chain, Hist etc. Their interface
quite battle-tested, I would strongly favour taking example from them, e.g.
type Tree
baseobj::CppValue{:TTree}
...
end
function Tree(name=nothing, title=nothing, model=nothing)
#like https://github.com/rootpy/rootpy/blob/master/rootpy/tree/tree.py#L886
end
In ROOT.jl I attempted to autogenerate the hierarchies using Clang.jl, but
it will never map properly I think.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jpata/ROOT.jl/issues/4#issuecomment-236197606, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABxEHwcw6INORNqarqTZ0oO0I-uI5Bxwks5qahAMgaJpZM4JQo5Q
.

@jpivarski this is certainly an idea I toyed with in desperation in early 2015, before I got the dynamic linking to work (what we do is dlopen julia in an exe dynamically linked with ROOT). For the record, using this kind of linking is fully functional, one just needs to recompile the julia REPL.
I thought about using Twisted, named pipes etc. I played around with it a bit but never got it to work well. You are probably much better prepared for this kind of an experiment than I was, though.
It seems kind of a nightmare though to have the two live in separate memory spaces so you can't exchange functions pointers, arrays without overhead. TTree-backed columnar analysis like the following would be much clunkier.
values = tree[cuts, "branch"] #possibly an expensive operation, generating a large array in memory
println(mean(values)) #julia function on ROOT-derived data

I do like the idea of julia not dying in case of ROOT segfaulting with this encapsulation. However, from the practical analysis world, it can be problematic forking in your jobs if you only want to be charged for the resources of 1 CPU.
It would be an additional layer of complexity. My experience with ROOT.jl (and CMSSW.jl) in a practical physics analysis was overall quite good, I had near native performance and good stability.

This separate-process interface wouldn't be as fast as direct linkage, but
it would be more flexible.

I have thought about this as well - and in fact, used to in the past to write ROOT files from Scala - but I don't think in this case it would be a good approach. The latency inherent in this model, for every function call, means that you need to use a programming API that supports queued/pipelined requests (e.g. promise/future) to get decent performance. And you need to users to understand how to properly use it. Our specific community is not used to this kind of programming model at all ... ;-)

Our specific community is not used to this kind of programming model at all ... ;-)

@jpivarski I think this is a crucial point, also about the question of any pushback inside HEP. It's good to change a few things for the better, but change too many things at once and nobody will use it. I'm not saying new is bad (I like Spark and @spawn), but we have to be careful to not be too far removed from how people are actually using ROOT: kind of a kitchen-sink Pandas/numpy/scipy equivalent.

the LLVM incompatibility
We're working on it.

@Keno: Thanks!! I can imagine that it's not a trivial problem at all.

@oschulz thinking about it more: we have a chance to get the API right, but it needs serious thought and also discussion with people who are not us.
shall we make a repo like JuliaHEP/API and have some markdown documents there? We can start to flesh out our ideas, then ask for feedback from other people who have undertaken similar efforts.

On Fri, Jul 29, 2016 at 1:07 PM, Joosep Pata<EMAIL_ADDRESS>wrote:

@oschulz https://github.com/oschulz thinking about it more: we have a
chance to get the API right, but it needs serious thought and also
discussion with people who are not us.
shall we make a repo like JuliaHEP/API and have some markdown documents
there? We can start to flesh out our ideas, then ask for feedback from
other people who have undertaken similar efforts.

I like that idea a lot. I'll contribute, but I'll also find people who have
physics deadlines to post their opinions as well. We might want to
categorize suggestions from computing-focused people (who know the
technical issues) and physics-focused people (who will be working with it,
day in, day out) differently. I know; some are both.
-- JIm

thanks @jpivarski. I certainly have some physics deadlines :D I should open an issue to myself.

done https://github.com/JuliaHEP/API/blob/master/ROOT.md

@jpata: thinking about it more: we have a chance to get the API right, but it needs serious thought and also discussion with people who are not us.

I fully agree. I'd like to get inputs from lots of people in the related fields - and on some issues, I hope we can also get advice from a Julia expert like Keno.

shall we make a repo like JuliaHEP/API and have some markdown documents there? We can start to flesh out our ideas,

I would actually prefer not to start with an abstract API spec, but with actual code: Prototyping with code, in my experience, often gives one a clearer idea of what work and what doesn't, than an abstract spec. Also, we already have some (I hope not too horrible) ROOT wrappers in ROOTFramework.jl, so I propose to use that as a starting point. We're already using that code in our (small) group here, and I need something that runs. However, we don't have a lot of code based on it yet, so I don't consider anything that's currently in ROOTFramework as sacred. For discussion and API proposals, I propose to use GitHub issues instead of a repo with Markdown docs - the issue system works well for such things, in my experience.
I mainly need TTree's and Histograms/Graphs at the moment, so these areas have a high priority for me, personally. I'm also currently in the lucky situation that I can officially devote some of my time to this. :-)

Sorry, I just saw that you created the ROOT.md already. I'd really like to convince you that issues will work better for this than documents, though. The Julia people use the GitHub issue system very effectively, for example, also for semi-permanent content like milestone checklists, etc. It makes it easier to discuss.

Like I said, I'm more than open to rename ROOTFramework to something else. Uh, and I'd also like a few days to clean it up a little, before we start in earnest. :-)

I'd say issues for discussion, .md for an status overview that we can easily show. I'm hesitant to write API code without a clear plan, having written already some useless APIs in the past. You're welcome to go ahead though. I'm not currently using julia for active work so my timeline is a bit different.
Prototypes are of course required.

@jpata: For more static content - like developing a user guide - why don't we just use a GitHub wiki (e.g. enable the wiki functionality for ROOTFramework.jl) instead a separate repo with markdown docs? I think a wiki would be a better fit. Then in the end, some of it can end up in the README and docs.

I'm hesitant to write API code without a clear plan, having written already some useless APIs in the past

I actually did quite a bit of thinking before I put code into ROOTFramework. :-) I'm not saying it's perfect - probably far from it - but the was some planning. :-)

Prototypes are of course required.

Very much so. Since we're not writing a pure Julia API here, the fact that we're wrapping C++ will drive (and has for me, so far) quite a few design decisions. That why I think it important to try things out (and even use them) while they are developed.

You're welcome to go ahead though. I'm not currently using julia for active work so my timeline is a bit different.

Ok, thanks. I'll go ahead and flesh ROOTFramework out a bit more, then, in the areas we need immediately. I'll try to quickly get the master to a state anyone can try out very (say two weeks) - if you can find some time (I know your're busy with your thesis), I'd be happy if you gave it a go. Please feel free to suggest (even drastic) changes, I really want to avoid tailoring this too much to our needs.
Uh, would you be Ok with removing the "API" repo in the mean time? As not a lot may happen on it, it'll be confusing to people who look into JuliaHEP.

I'd keep the API it for the moment (if you think it's confusing I'll move it under my own name). But however smart any one of us are and how much thinking and designing we do, we are not smarter that the hivemind of ROOT users, julia users, rootpy developers etc.
If you develop something now, in my experience, it will be harder to change in the future. For example, all ROOTDataFrames, ROOTHistograms etc are at the moment useless.
Strong vote for design document.

Okay, another weird suggestion: as part of my DIANA-HEP work, I'm going to try to solicit opinions about analysis script APIs (generally) in a focus group a few weeks from now. If that works well, future focus groups could be about the Julia-ROOT interface. We could prepare several mock-up APIs, convince ourselves that they'll work, and then present them to a group of physicists with laptops to try them all out and say what they like/don't like about them. That's pretty close to how focus groups work in industry, replacing "data analysis API" for "brand of soap."
In that case, our job would be to come up with a basis set of trial APIs--- they don't have to have working machinery behind them, just enough for participants to get the "feel" of them. It should be a basis set--- covering the space of possibilities with as few vectors as possible to avoid overwhelming the participants with nearly identical choices. For our own development, the design documents can be private, and I find Google Docs to be a very responsive way to collaboratively edit.
What do you think?

Focus group sounds really good. Given that I've been doing HEP analysis for some years, I have also some feeling for what works and what doesn't, but I'd like to take the average over some ensemble. For many questions, we might even get an answer from the root devs.
Trial APIs are also good, though I'm not sure we can have several meaningful ones very quickly. Maybe one and iterate on that.
Private google docs I'm less a fan of. In case dev discussions are held in private, they will be lost for future collaborators, who will keep offering the same old ideas. I think anybody should be able to trace why some design choice was adopted.
I'm reluctant to turn this into an all-talk and no-action kind of gig, so I propose we think a bit, write some code, and then discuss again in a couple of days.

I'd keep the API it for the moment (if you think it's confusing I'll move it under my own name).

Would you, at least for the moment? Like I said, I don't want to do ROOTFramework (or whatever it's name will be) by myself, and I'm more than open to any suggestions concerning it's current (very limited) API. However, I think I'll have it in a state that people can actually use for simple things soon, and having API with different/future API concepts right next to it may be confusing.

Done.
I strongly believe that a solution that we code away in the next weeks with thought but without discussion will result in a local optimum, but not a global one.
We must use the experience of other devs out there, otherwise it's unlikely we'll get something much better than we already have.
I'm also closing this issue as it's no longer about the technicalities of using ROOT with Cxx.jl.

We must use the experience of other devs out there, otherwise it's unlikely we'll get something much better than we already have.

I fully agree!

A heretical idea occurred to me about accessing ROOT in Cxx.jl: what about ROOT 5, rather than ROOT 6? It was based on CINT, not LLVM, and many analyses are still using it. I know it's not a good idea to depend on obsolete software, but the compatibility between ROOT and Julia could be advertised as "ROOT 5 or ROOT 6.x.y" where "6.x.y" is the version with the LLVM issues resolved.

ROOT 5 is a no go, an unmaintained and buggy evolutionary dead end.
I haven't had a lot of time now to work on this, I think as Julia 0.5 and cxx are released and stable, this project just needs someone elbow grease to flesh out the essentials in Cxx as @oschulz has started.

I apologize if this information is available and I just can't find it, but can an appropriately-built version of ROOT and Julia be found anywhere? There was some talk earlier of putting it on CVMFS. Has this happened?
I want to do a tutorial session with some physicists at the LPC (cmslpc-sl6.fnal.gov or lxplus.cern.ch) to convince them that Julia is interesting/useful for their analyses. Without any way to access data in ROOT files, it's a hard sell. So we don't need the frameworks to be fully worked out, such as API differences between ROOT.jl and ROOTFramework.jl: we just need enough compatibility to load data into Julia data structures so that I can show them Julia. If ROOT 5 works, I'd use that. (I'm trying it now and am running into compilation errors.)
If the physics audience gets hooked on the idea of doing analysis in Julia and then find that the Julia-ROOT link is not production ready (either because of the custom build or ROOT 5), that's okay. They'll demand compatibility in a future ROOT release. In fact, the existence of vocal users might be necessary to get this incompatibility resolved.
So let's get these eggs laying chickens! (Or chickens laying eggs, whichever...)

Pere Mato Vila was working on the julia+ROOT cvmfs install, but they've different constraints at CERN for installing software and it's not high on their priority list I suppose.
Did you try setting it up based on this repo? It works under OSX and linux when I last tried, there's also a docker image: https://github.com/jpata/docker-julia-root-cc7/blob/master/Dockerfile
There is no way to install on a non-source julia. I currently have no time to invest in making a community around this project, but if you have some people interested, that would be great.
I'd be happy if you and @oschulz exercised commit rights to this project under the JuliaHEP org.

I'm trying to create the community interest, and for that I'll need a minimally working version. If you have a recipe, I'll build an Amazon AMI and give everyone in the tutorial access to log in and try it. Maybe as many instances as there are participants.
This repo says you're changing things around for Cxx.jl, so don't expect it to work. Is there a branch or a revision number that I can Pkg.clone?

@jpivarski:

A heretical idea occurred to me about accessing ROOT in Cxx.jl: what about ROOT 5

I'm actually using ROOT-5 at the moment, without problems, because of the LLVM conflict between Julia and ROOT-6 (I haven't managed to get @jpata's workaround to run on my system, but I'll try again). Joseep is right, though, of course: ROOT-5 is a dead and, and won't be maintained for much longer (if it's still supported at all).

I haven't found time in the last weeks to work on the ROOT/Cxx.jl integration, but I hope to make some progress over the next two weeks.

I overheard some conversations about a ROOT-LLVM incompatibility due to a symbol clash with LLVM loaded by a graphics library. This is the bug report:
https://sft.its.cern.ch/jira/browse/ROOT-7821
I'm told it was fixed in 6.06/00, but only for CMake builds, not for configure builds. When you encountered the Julia-ROOT incompatibility, were you using this patch?

@jpivarski: I'm not sure it's the same issue we're running into, but I'll try!

Yes, some newer OpenGL also uses it's own llvm and I've seen this clash as well. ROOT specs simply expect to call TCling::InitInterpreter before referring to any other external LLVM, in which case ROOT can hide itself. This is what I've done in ROOT.jl by recompiling the julia binary.
You can see the dockerfile with the ROOT version here: https://github.com/jpata/docker-julia-root-cc7/blob/master/Dockerfile#L11, so most likely this is already included and didn't fix the fundamental incompatibility.
I've thought about it over the past months and I believe it's not worth it to port ROOT, it's probably better for HEP to go cold turkey.

@jpata: I believe it's not worth it to port ROOT, it's probably better for HEP to go cold turkey.
I agree that it would be best to do as many things as possible in pure Julia, integrated with the rest of the Julia ecosystem. Unfortunately, it's often still unavoidable to read and write root files, to be able to work with one's community. Also, ROOT plots are often expected or even mandated.
Just curious, what's the current status with ROOT.jl - meaning, rjulia. Does it works reliably on Linux now? If so, it would really be nice to be able to build it separately from Julia, now that Cxx.jl can be build separately (e.g. with official Julia binaries) too.

When C++ compatibility is mature in Julia, it will seem like an odd story that it's compatible with all C++ libraries "except ROOT." ROOT is quintessentially C++, and it's ironic that ROOT and Julia are incompatible precisely because they took the same route to JIT.
But even if the direct route to compatibility is blocked by (what most would consider) a weird bug, there are other avenues. For histogram-like data, Histogrammar is a go-between for everything, including the much harder to bridge C++-Java hurdle. For ntuple-like data, I'm working on (and looking for help with) exporting ROOT TTrees to Apache Arrow, which can maintain all the deep structure of a TTree and zero-copy between libraries and languages sharing the same process. I see there are some rumblings of using Arrow in Julia: http://julia-programming-language.2336112.n4.nabble.com/Apache-Arrow-Feather-td38644.html

what's the current status with ROOT.jl - meaning, rjulia. Does it works reliably on Linux now? If so, it would really be nice to be able to build it separately

@jpata: Sorry, I hadn't realized that this is already possible. I just looked at your current docker-julia-root-cc7, and saw that building the modified julia executable simple, and just needs a Julia installation (and not the full Julia source tree - I did need it in the past, for some Makefiles, if I recall correctly?).
I built the container, and tried my ROOTFramework (dev branch) in it - at least the examples run fine out of the box. Very nice!
However, on Ubuntu (self-compiled Julia bin-dist, self-compiled ROOT) things crash as soon as I load ROOT's libGui.so. I can load libHist.so, though. No idea why -  but the fact that it runs in your Docker container is very encouraging.

When C++ compatibility is mature in Julia

@jpivarski: I'd say it's already pretty mature. At least Cxx.jl is a registered package now, and it hasn't let me down yet. :-)

For ntuple-like data, I'm working on (and looking for help with) exporting ROOT TTrees to Apache Arrow

Arrow would make for a nice common in-memory representation - but from what I understand, that would still require something that can read ROOT files (i.e. ROOT) and Julia to run together, share memory and communicate in some fashion, right?
Apache Parquet would make for a nice on-disk alternative to ROOT files with TTrees - however, I don't think there's Julia support for Parquet yet (which would be very nice thing to have in general, not only for HEP).
The problem is, translation efforts (at least automatic ones) get in serious trouble for files that store custom classes (maybe even using custom streamers) in TTrees. In the end, I don't think a large part of the HEP community will really accept Julia if there's no way to at least read ROOT files with histograms and trees - because Python can. But I think we're getting there: With ROOT-5 there doesn't seem to be a problem, @jpata's modified Julia exe looks like a viable workaround for ROOT-6, and long term, I trust Keno and the ROOT team to find a clean solution.

As i understand it, the Cxx.jl package requires Julia 0.5, which is not yet the production release. That's all I meant.
There is a translation effort from concrete C++ data types into abstract types. My rootconverter GitHub project is an early example of that: I made a table of many-to-one mappings, such as "char*, std::string, and TString all go to Avro strings" and "array, std::vector, TObjectArray, and TClonesArray all go to Avro arrays," etc. Pointers were an interesting case, because although they could be used for many different things, a C++ user is likely to use them to mean a nullable type, so I mapped them to Avro's equivalent of a maybe monad.
This kind of recursive transformation can be automated (root2avro doesn't have to be given a schema: it gets its type information from ROOT's TStreamerInfo). The only problem is that I have no idea how exhaustive it is. Users may be discovering new cases for years to come.
However, it does stand as an illustration of how this kind of conversion can be automated, not just into Avro, but Arrow and therefore many other formats.
To share Arrow data in memory, there may be some way to do it with interprocess communication, like a RamFS or streaming chunks from one process to another. At worst, there are file formats like Feather or Parquet, which could be streamed. I thought this would be necessary for hard-to-bridge platforms like Java; I'm surprised we have to resort to it for Julia.
(Surprised in retrospect: I know the issues.)

As i understand it, the Cxx.jl package requires Julia 0.5, which is not yet the production release. That's all I meant.

Julia 0.5 is officially the current release now. Of course it's likely that there will still be a few breaking changes in 0.6 and 1.0.

There is a translation effort from concrete C++ data types [...] Pointers were an interesting case [...] a C++ user is likely to use them to mean a nullable type

Unfortunately, the other use of pointers is for polymorphic types. As ROOT can store true nested C++ object structures, the exact "type" of a pointer-style field isn't necessarily fixed, as it can be a subtype. I have encountered this in real-world ROOT data. Whether it is wise to do something like this is questionable, of course (I wouldn't) - but usually, one can't choose one's data source. I fear such things will be hard to map to Avro/Parquet (which are excellent formats).

root2avro doesn't have to be given a schema: it gets its type information from ROOT's TStreamerInfo

I thought about something like this in the past - I think this is great. Are you planning to support Parquet, too? It's probably a better fit for most ROOT-style data and the related use cases than Avro, due to Parquet's columnar structure.

Class inheritance can be expressed in Avro/Parquet by a tagged union of all concrete classes in the tree. However, I don't think that they can be stored in a ROOT TTree. It would have to somehow include an indicator of the concrete class along with the instance data.
And yes, the goal is to map ROOT TTree data to any output format through Arrow, assuming that Arrow gets enough file format support. Avro, Parquet, and Feather at the very least. Maybe Numpy for flat TNtuples.

Class inheritance can be expressed in Avro/Parquet by a tagged union of all concrete classes in the tree. However, I don't think that they can be stored in a ROOT TTree. It would have to somehow include an indicator of the concrete class along with the instance data.
And yes, the goal is to map ROOT TTree data to any output format through Arrow, assuming that Arrow gets enough file format support. Avro, Parquet, and Feather at the very least. Maybe Numpy for flat TNtuples.

Class inheritance can be expressed in Avro/Parquet by a tagged union of all concrete classes in the tree.

True, it's possible - not not easy, I would imagine. One needs to know all possible classes up front.

However, I don't think that they can be stored in a ROOT TTree. It would have to somehow include an indicator of the concrete class along with the instance data.

To my knowledge, they can be, and instance type information is stored.

goal is to map ROOT TTree data to any output format through Arrow, assuming that Arrow gets enough file format support. Avro, Parquet, and Feather [...]

That would be awesome! Of course, we'll need good Julia bindings, too. :-)

dockerfile with the ROOT version here: https://github.com/jpata/docker-julia-root-cc7/blob/master/Dockerfile#L11

@jpata: I played around with this a bit, and then tried without compiling Julia - and it works! I was able - using the official Julia and ROOT binaries - to compile Cxx.jl and ROOT.jl's julia binary and run the ROOTFramework (dev branch) examples. The resulting container image is quite small (2.4 GB total) and builds fairly quickly (only Cxx.jl needs to be compiled).
I'll create a proper Dockerfile with all the steps.


[BUG] Latex in als_movielens_diversity_metrics.ipynb not rendering properly

Description

The formulas are not rendering in the notebook: https://github.com/microsoft/recommenders/blob/staging/examples/03_evaluate/als_movielens_diversity_metrics.ipynb
I can see the code with $$
In which platform does it happen?

How do we replicate the issue?

Expected behavior (i.e. solution)

Other Comments
FYI @anargyri @YanZhangADS

This is strange, because it shows well on Jupyter or VSCode.
It is more general in fact, also other notebooks have the same issue e.g. https://github.com/microsoft/recommenders/blob/staging/examples/02_model_collaborative_filtering/sar_deep_dive.ipynb
I recall the formulas used to render properly in the SAR notebook until recently.

Dup of https://github.com/microsoft/recommenders/issues/1567


Server loader with cache headers can't cache first load

Describe the bug
If I have cache headers set in my server load function
export async function load(params) {
	console.log('LOADING');

	// const val = params.url.searchParams.get('val');
	const val = params.url.search;

	const result = {
		searchProps: 'Value ' + val + ' ' + ' CACHE TEST VALUE === ' + i++
	};

	params.setHeaders({
		'cache-control': 'max-age=60'
		//Vary: 'Cookie'
	});

	return result;
}

the headers work, and the page is cached, but if I start on this page, browse elsewhere, and then hit the back button, the loader re-runs.
The reason appears to be that the initial load of my site renders the entire page (which does have the correct headers), but subsequent navigations fetch a __data.js with the right url attached, so browsing back to it fetches that, which isn't (yet) cached.
Reproduction
The minimal repro is here: https://github.com/arackaf/booklist/tree/special/bug-repro/svelte-kit
the directory above that has a huge project around it, but that directory I've linked to contains a minimal SvelteKit app that reproduces this (ostensible) bug.
It's currently deployed here: https://booklist-svelte-kit.vercel.app/
As you navigate, you'll see fresh calls to fetch __data.json, even when you click back to the starting page, which, again, is not (yet) cached.
Logs
No response
System Info
System:
    OS: macOS 12.6
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
    Memory: 149.78 MB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 16.14.2 - /usr/local/Cellar/nvm/0.38.0/versions/node/v16.14.2/bin/node
    Yarn: 1.22.17 - /usr/local/Cellar/nvm/0.38.0/versions/node/v16.11.1/bin/yarn
    npm: 8.18.0 - ~/Documents/node_modules/.bin/npm
  Browsers:
    Chrome: 107.0.5304.110
    Firefox: 107.0
  npmPackages:
    @sveltejs/adapter-auto: next => 1.0.0-next.89
    @sveltejs/kit: next => 1.0.0-next.557
    svelte: ^3.44.0 => 3.53.1
    vite: ^3.1.0 => 3.2.4

Severity
serious, but I can work around it
Additional Information
No response

This sounds like #4625, which is supposed to have been already fixed in #6461, but perhaps not.

It's related to #4625, but that was only about shared load functions. We would need to add a similar-but-different mechanism for the __data.json case.

@dummdidumm and as I've been discussing with Rich, you'd probably need to NOT serve cached results on subsequent load if a custom vary header were present (or else find some creative way to respect the vary header ... ie record all the Vary'd values along side the ttl, and check them on second request)